ABSTRACT

Providing feedback to learners on their writing assignments is perhaps one of the most important and time-consuming tasks that a supervisor performs. In e-Education environments, giving feedback becomes more challenging because there are often no possibilities for face-to-face discussions with learners. Typically, a supervisor provides comments to learners in written form via email; however, the use of recorded audio feedback (RAF) in e-Education environments has become a viable alternative. The purpose of this case study was to examine learners’ perceptions of RAF and written feedback for their assignments at the University of Jyväskylä (Finland) and at Keio University SFC (Japan). Formative feedback was used to study RAF. Data were collected through surveys and interviews. The results show that learners tend to have positive feelings toward RAF. The findings also indicate that learners’ methods of revising their assignments based on the feedback they receive may impact their preference for one modality over the other. We introduce a process model based on the findings of our study that highlights RAF best practices and guides supervisors in effective use of RAF.

Keywords:

Formative Feedback, Recorded Audio Feedback, RAF, Cross-Cultural Higher Education, e-Education Environments, Cultural Dimensions

1. Introduction

The multicultural nature of higher education makes it critical that supervisors and instructional designers, especially those working in e-Education environments, are culturally aware when they deliver instruction, feedback and assessment to learners. The increasing use of technology-based e-Education environments and tools in higher education promotes the development of new approaches to enhance the methods and the quality of feedback given to learners (Cavanaugh & Song, 2014) . An e-Education environment is defined as a set of information and communication technology (ICT) and web-based teaching and learning tools designed to enhance users’ learning experience in the learning process. Throughout this study, the term “learner” refers to the person receiving the feedback, while the term “supervisor” refers to the individual giving the feedback. Our study is within the scope of cross-cultural pedagogical research, which focuses on teaching and learning processes in cross-cultural environments.

Providing feedback is an essential part of the teaching and learning process and can be utilised by learners to enhance their future academic performance (Molloy & Bound, 2013; Taras, 2005) . Effective feedback needs to 1) explain what progress is being made toward the study goal or study objective, 2) explain how the learner has performed and c) provide advice to help the learner improve (Nicol & Macfarlane-Dick, 2006; Schwartz & Gurung, 2012) . Formative feedback concerns development, improvement and learning (Taras, 2005) . Summative feedback concerns accountability and performance (Taras, 2005) . Summative feedback evaluates a person’s learning at the end of an instructional unit by comparing it against some standard or benchmark.

Recorded audio feedback (RAF) is one method of providing feedback that is becoming increasingly popular. RAF can be defined as formative and/or summative messages that are recorded and distributed by supervisors as digital audio files to individual learners or learner groups in response to both on-going and submitted work (Middleton & Nortcliffe, 2010) . Researchers are still evaluating this feedback delivery in a higher education context; thus, RAF needs further evidence-based pedagogical research and modelling, especially in e-Education environments (Schwartz & Gurung, 2012) .

The purpose of our study is to examine distance learners’ perspectives on RAF and written feedback in a three-phase reporting task on a software development related e-course entitled “Requirements Engineering” (RE) at the University of Jyväskylä. The paper examines the extent to which learners respond to and engage with RAF. In addition, we carried out a study at Keio University SFC (Japan) among international learners in order to examine the cultural dimensions in the RAF context. Based on our findings, we introduce a process model that highlights RAF best practices and guides supervisors in effectively utilising RAF.

The remainder of the paper is organised as follows. In Section 2, we conduct a literature review of related work. In Section 3, we describe our theoretical framework. In Section 4, we present categories of RAF. In Section 5, we describe our study. In Section 6, we introduce our findings and a process model. In Section 7, we conclude and describe some ideas for future research.

2. Related Work

Several studies on feedback show the importance of the timeliness of the feedback (Nicol & Macfarlane-Dick, 2006; Archer, 2010; Butler, Godbole, & Marsh, 2013; Nicol, 2010; Rudland, Wilkinson, Wearn, Nicol, Tunny, Owen, & O’Keefe, 2013; Stern & Solomon, 2006) . Learners highlighted the impact of feedback timeliness on future performance. Further, the clarity of written comments was considered to be an important factor among learners. In practice, written comments on assignments are often unclear, difficult to read and confusing to learners. In many cases, written comments contain academic jargon that is unfamiliar or unclear to the learner.

From the learner’s viewpoint, good-quality feedback contains detailed information on how to improve, is applicable to future work, is personal and is timely (Nicol & Macfarlane-Dick, 2006) . RAF can help learners to overcome the issue of clarity. Learners may benefit more from RAF because listening to the voice of the supervisor is more appealing than just reading his or her comments, being able to hear the comments while reading the document is more personal and the feedback itself is clearer. However, some studies have found that learners find audio comments challenging when they try to locate which comment belongs to which problem in the assignment. Written commentary makes it easier to locate the problem. Although several studies have been carried out on RAF, they were mostly conducted in class courses, in which it is not clear if revised versions of the learners’ assignments were involved. Further, most studies were done in face-to-face class contexts, in which face-to-face dialogue with the supervisor can help to clarify RAF comments (Cavanaugh & Song, 2014; Stern & Solomon, 2006; Cann, 2014; Cavanaugh & Song, 2015; Chew, 2014; Eckhouse & Carroll, 2013) . Overall in these cases, the impact of RAF on the learning process is unclear. Our contribution to RAF research is to focus on distance learners’ perspectives on RAF in a higher e-Education setting.

Current scholarship indicates that learners and supervisors have an overall positive attitude toward RAF. Studies by Merry and Orsmond (Merry & Orsmond, 2008) and Nortcliff and Middleton (Nortcliffe & Middleton, 2008) measured the effectiveness of supervisors’ RAF and concluded that learners perceived and implemented audio file feedback in different and more meaningful ways than they did written feedback. Audio feedback has also been noted as bridging a gap between the learner and the supervisor and being a time-saver for the supervisor (Schwartz & Gurung, 2012; Gould & Day, 2013; Ice, Curtis, Phillips, & Wells, 2007; Lunt & Curran, 2010; Munro & Hollingworth, 2014; McCarthy, 2015) .

The technologies and devices available to compose and deliver RAF have advanced tremendously during the last five years. Several e-Education environments, such as Optima (Optima, 2017) , support RAF. Programs like Adobe Acrobat Pro and Microsoft Word also make it possible to embed RAF files into different parts of an assignment. Separate audio files (.ogg or.mp3) can be created, for example, with Audacity, which is a free, open source, cross-platform audio software for multi-track recording and editing.

In our study, we use the web-based e-Education environment Optima (Optima, 2017) , which is widely used at the University of Jyväskylä, to compose and deliver RAF to individual learners and learner groups. Specifically, we focused on a distance-learning course in the higher education context where learners do not have the opportunity to engage in synchronous dialogue with their supervisor.

3. Theoretical Framework

This study is informed by Kolb’s learning theory (Kolb, 2014) , the progressive inquiry learning model (Hakkarainen, Lonka, & Lipponen, 2004; Hakkarainen, 2003) and cultural dimensions of e-Education environments (Parrish & Linder- VanBerschot, 2010; Heimbürger, 2016; Heimbürger, 2017) .

3.1. Kolb’s Learning Theory

According to Kolb’s learning theory (Kolb, 2014) , effective learning occurs when a person progresses through a cycle of four stages: 1) having a concrete experience followed by 2) observing and reflecting on that experience. This leads to 3) the formation of abstract concepts (analysis) and generalisations (conclusions) that are then 4) used to test hypotheses in future situations, resulting in new experiences (Figure 1). In concrete experience, a new experience of a situation or a reinterpretation of an existing experience is encountered. In reflective observation (of the new experience), of particular importance are any inconsistencies between experience and understanding. In abstract conceptualisation, reflection

gives rise to a new idea or a modification of an existing abstract concept. In active experimentation, the learner can apply a new idea to the world around him or her to see what happens. Learning is the process whereby knowledge is created through the transformation of experience.

Some studies suggest that RAF may play an important role in Kolb’s learning cycle, allowing learners to reflect on their experiences and use feedback from supervisors and peers to process, take ownership of and integrate their new ideas into future assignments (Chew, 2014; Trimingham & Simmons, 2010) .

3.2. Progressive Inquiry Learning Model

The progressive inquiry (PI) model is a pedagogical model that facilitates the same kind of productive knowledge practices of working with knowledge in higher education that characterise scientific research communities (Hakkarainen, Lonka, & Lipponen, 2004; Hakkarainen, 2003; Crumly, 2014) . It emphasises shared expertise and collaborative work for knowledge building and inquiry by setting up the context via questions, explanations, theories and scientific information in a cycle of deepening inquiry. The PI model starts from a problem that calls for an explanation, the inquiry process is socially distributed and then the inquiry gradually progresses from a working theory toward more specific questions. The PI model is often used with computer-supported collaborative learning in e-Education environments. We apply the PI model in our RE course because it pedagogically supports the course’s learning objectives, structure and study phases.

In a progressive inquiry process (Figure 2), the supervisor creates a context for inquiry by presenting a multidisciplinary approach to a theoretical or real- life phenomenon. After this, the learners start defining their own questions and intuitive working theories about it. Learners’ questions and explanations are shared and evaluated together with the supervisor and their peers. The evaluation directs how they use authoritative information sources, the iterative elaboration of subordinate study questions and more advanced theories and explanations.

3.3. Cultural Dimensions in Cross-Cultural e-Education

Culture is embodied in how people interact with other individuals and with their environment; it is a way of life formed under specific historical, natural and social conditions (Wang, 2009) . Human nature comprises the assumed commonalities that all humans share because they are members of the same species― Homo sapiens (Parrish & Linder-VanBerschot, 2010) (Figure 3). Parrish and Linder-Van Berschot (Parrish & Linder-VanBerschot, 2010) outlined eight cultural dimensions in the higher education context. These dimensions can be divided into three main categories that are most likely to impact educational situations and have an effect on the learning process:

・ social relationships: 1) equality and authority, 2) individualism and collectivism, 3) nurture and challenge;

・ epistemological beliefs: 4) stability-seeking and uncertainty acceptance, 5) logic argumentation and rationality, 6) causality and complex systems; and

・ temporal perceptions: 7) clock and event time, 8) linear and cyclical time.

The cultural dimensions in (Parrish & Linder-VanBerschot, 2010) are imposed on three main cultural models: Hall’s model (Hall, 1990) , Hofstede’s model (Hofstede, Hofstede, & Minkov 2010) and Lewis’ model (Lewis, 2005) . From the learner’s point of view, it is essential to understand that software development and software industry are nowadays mainly realised by multicultural teams in enterprises, and working in these teams requires cultural awareness. From the supervisor’s point of view, the learners in the RE course are coming from different cultures, and this should be taken into account when delivering RAF in an e-Education environment.

4. Categories of RAF

Three principles for effective formative feedback have been presented by Stern and Solomon (Stern & Solomon, 2006) : 1) provide positive comments in addition to corrections, 2) provide feedback only on a few select areas that are very essential for the writing assignment in question that support the individual learner’s or group of learners’ goals and 3) provide comments that identify patterns

of weaknesses, strengths and errors. Stern and Solomon identified four feedback levels: global, middle, micro and meta. We apply this categorisation to our RAF study as follows:

・ Global-level RAF includes comments that look at the work and writing as a whole. Examples of this include overall quality, paper/report structure, organisation, creativity and the learner’s or learner group’s voice.

・ Middle-level RAF includes comments that focus on ideas and evidence supporting them as well as how the ideas were expressed at the paragraph/sentence level. Examples of this include evaluating the quality of specific thoughts and claims, procedure and technique, support or evidence for claims, request for content clarification and paragraph/sentence structure and style.

・ Micro-level RAF includes all the basic technical issues of writing and referencing sources. Examples of this include word choice and phrasing, missing words and pieces, grammar and punctuation, spelling and typos, technical style, references and citations

・ Meta-level RAF includes comments such as the supervisor’s invitation to discuss the work, scientific advice (e.g. about related research organisations) and following the given template as well as supervisor’s comments on the students’ patterns of errors.

Formative feedback also addresses how well supervisors are doing and it provides the data back to supervisors. The feedback data can be reviewed and adjustments can be done according to it, and also supervisors can enjoy identified successes. Formative feedback is intended to help improve learners’ learning processes.

5. Study

5.1. Context

Our case study had two contexts, one at the University of Jyväskylä (JYU) in the Faculty of Information Technology and one at the Keio University SFC (Shonan Fujisawa Campus). The case course at JYU was an advanced level course in requirements engineering (RE). The author of the paper is serving as the supervising teacher in the RE course, and she has already 10 years’ experience of developing and managing the RE course. Keio University SFC and University of Jyväskylä have the memorandum of understanding (MoU) agreement in collaborative research and education. The author of the paper is the key person of the MoU at JYU site and her colleague, who implemented the RAF study at Keio, is the key person of the MoU at Keio site. One leading theme of our collaboration is cross-cultural research and education.

A requirement in the software development context is defined as a condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification or other formally imposed document (Pohl, 2010) . A well-formed requirement is a statement of system capability that must be met or possessed by a system to satisfy user needs or objectives and that is qualified by measurable conditions and bounded by constraints (Pohl, 2010) . RE contains a set of activities for discovering, analysing, documenting, validating and maintaining a set of system requirements. It is divided into two main groups of activities: requirements development and requirements management. Requirements development includes activities related to discovering, analysing, documenting and validating requirements, whereas requirements management includes activities related to maintenance, status tracking, traceability and change management of requirements.

The RE course in the Faculty of Information Technology at the University of Jyväskylä is a web-based course implemented in Optima’s e-Education system. Optima is one of the e-Education platforms used and supported at the University of Jyväskylä. The supervisor designs and implements the Optima working space for the course and opens it up to learners. Learners can submit their assignment to a return box in the course’s working space. The supervisor has monitoring and overview functions and tools in the course’s space, which makes following and evaluating assignments easy. Learners see only their own returns and supervisor’s evaluations. The supervisor prepares the RAF files with Optima’s new object/create audio file function and defines the access rights to a certain learner or a learner group. When the RAF is ready, an email is sent to a learner or a learner group concerned.

Progressive inquiry is applied in the course as a pedagogical model (see Section 3.2). The RE course consists of three phases. Each phase has a theoretical and practical component, and the three phases form a cumulative process. After each phase, learners submit an assignment and the supervisor evaluates it and gives feedback on how to proceed. In our study, the feedback was given as follows:

・ Phase 1: Feedback in written form by email

・ Phase 2: Feedback in written form by email and RAF

・ Phase 3: Feedback only by RAF.

The case reporting assignment “Global Environmental Policy Management” at Keio University SFC was related to the Global Environmental Systems Leaders Programme (GESL).

5.2. Participants

The first part of the study was conducted in an advanced RE course in the Faculty of Information Technology at the University of Jyväskylä during 2015-2017. The RE course was realised as a web-based distance learning course in an Optima e-Education environment. The principal instructor focuses on supervising learners during the course, here called “supervising teaching”. Learners were invited to participate in the email survey for the RAF study, and a total of 90% (45/50) completed the survey. The second part of the study was conducted at Keio University SFC (Japan), where 19 learners completed the survey (12 learners from the Global Environmental Systems Leaders Programme and 7 learners from the Multi Database Laboratory). At Keio University SFC, the RAF survey was distributed as a Google Form.

5.3. Questionnaire

The main aim of the survey was to identify and suggest best practices in terms of expectations around using RAF in higher education. The aims of this questionnaire study were as follows:

・ to improve learners’ engagement with feedback

・ to make more effective use of staff time

・ to investigate simple and efficient systems for delivering RAF

・ to identify influential cultural dimensions in the context of RAF

In addition to structured questions, free-form questions about the technical development of RAF were distributed to learners at Jyväskylä University. This was done because all of the learners were studying computer science, and they were interested in developing ICT systems.

6. Findings

In both studies, most of the learners had a positive experience with RAF. With RAF, more detailed feedback can be given, which learners can then listen to more than once and in their own time. They saw RAF as more personal than written comments. They felt that RAF was easier to understand, more detailed, more motivational and more personal. Around 60% of the learners preferred a combination of RAF and only main points written, and 40% thought that RAF feedback alone was sufficient. According to learners, combination feedback should consist of written main points (in the form of a bullet list) sent via email and a more detailed explanation via RAF. They stated that the suitable length of the RAF was 5 - 10 minutes. Both one audio file clearly indicating the points in the document (for example: page three, second paragraph, third line) and several audio files embedded in the PDF document in various locations were considered to be equally good options.

Learners were also asked to identify what they considered to be the three most important issues with RAF. The following options were given: 1) identifying errors, 2) giving praise, 3) correcting errors, 4) explaining misunderstandings, 5) demonstrating correct practices, 6) engaging learners in thinking, 7) suggesting further reading, 8) asking learners to justify their solution(s) in their assignments and 9) suggesting approaches for future assignments. They rated explaining misunderstandings, demonstrating correct practices and suggesting approaches for future assignments as the most important topics. We also asked them to describe the best way to structure RAF. The following options were given: 1) paragraph-by-paragraph comments, 2) general comments concerning the whole document, 3) section-by-section comments, 4) summary of the next steps, 5) appreciations and 6) critical comments. According to the learners’ answers, the best structure is as follows: 1) general comments concerning the whole document, 2) section/paragraph-by-section/paragraph comments, 3) summary of the next steps, 4) critical comments, and 5) appreciations. Concerning the supervisor’s tone of voice in the RAF, the learners most commonly chose the personal (rich nuances) option over the formal (very polite) or natural (no expression of feelings) option. Also, around 80% of the learners thought that the tone of the supervisor’s voice could convey whether the changes to be made were major or minor.

More statements in the RAF survey and their per cent (%) support are presented in Table 1. The options were strongly agree (SA), agree (A), neutral (N), disagree (D) and strongly disagree (SD).

The survey also included seven statements each one with two options measuring cultural dimensions (see Section 3.3). The test groups included learners from Finland (45) and Asia (19) (Japan, Malaysia, Thailand, Indonesia and Vietnam). The first statement included the following options: “In the RAF, the supervisor states exactly how you should correct your assignment”, and “In the RAF, the supervisor proposes how you could improve your assignment”. These options were equally supported by the learners.

The second statement included options: “After you have listened to the RAF, you adjust to the supervisor’s comments”, and “After you have listened to the RAF, you would like to express your own point of view”. These options were also equally supported by the learners. Both statements were related to the concept of authority in cultural models (see Section 3.3).

The third statement revealed some differences between group-oriented and individual-oriented societies, another cultural dimension measured in cultural models: “In group RAF, praise will be given to the whole group” was supported

by 74% of the learners. “In group RAF, praise will be given to the best learner” was supported by 26% of learners.

The fourth statement: “In learning activities, the focus is on getting the right answer, ambiguity is to be avoided, supervisors are expected to have right answers” was supported by 21% of the learners, whereas the statement “In learning activities, the focus is more open-ended, like discussions and project work, ambiguity is a natural condition, and the supervisor can say, ‘I don’t know’” was supported by 79% of the learners. This issue indicated some minor differences between attitudes toward authority in Finland versus in the Asian countries.

In the fifth statement options (a) and (b) were equally supported: (a) “In learning activities; there is a focus on logical argumentation to find truth and an insistence on single truths based on logical reasoning; debate and argumentation are learning activities; being right is the most important” and (b) “In learning activities, there is a focus on achieving practical and socially acceptable outcomes and an acceptance of multiple truths based on experience; consensus building is a learning activity; being virtuous is the most important”. No significant cultural differences occurred in these answers.

The sixth statement showed differences between Finnish learners (cause-effect) and Asian learners (situational). The statement “Learners are expected to be goal-oriented; knowledge is tied to “cause-and-effect” explanations and there is a focus on stable knowledge and rules” was mostly supported by Finnish learners. The statement “There is more willingness to work within situational constraints; knowledge is tied to explanations of systems and situations and there is a focus on evolving and situational knowledge” was mostly supported by Asian learners.

The same situation arose between the options of the seventh statement related to the concept of time. The statement (representing a linear time concept) “Time is to be managed; learning proceeds along a linear path with clear prerequisites and milestones; goal-setting is essential to learning; opportunities are not to be wasted; chances do not present themselves twice; the past is irrelevant and future goals are important” was mostly supported by Finnish learners. The statement (representing a cyclical time concept) “One adapts to time; learning is seen as practice towards slowly increasing perfection; goals are secondary; one adapts to the situation to draw from it as much as possible; time exists for observation and reflection; rushing is counter-productive to achievement because time is a series of cycles; opportunities recur, and when they do, one may make wiser decisions; the past is influential because cycles repeat; one carries the past forward; repetition is valuable for learning” was mostly supported by Asian learners.

Related to cultural issues, the results seemed slightly to follow the cultural dimensions presented in Parrish & Linder-VanBerschot study (see Section 3.3) (Parrish & Linder-VanBerschot, 2010) . However, generalisations cannot be made based on this study because of its small scale and also because the Asian test group included people from several different Asian countries. We found that the supervisor’s tone of voice was an important issue and was dependent on culture. Overall, although the content of RAF can be culturally neutral, the supervisor’s speaking style could be culturally adaptable.

Based on our findings we created a RAF process model highlighting the main actions for preparing RAF in an e-Education context (Figure 4).

7. Conclusions and Issues for Further Research

Our study examined the use of RAF in higher e-Education especially in distance learning context. It found that RAF had certain advantages over written feedback. With RAF, supervisors can use clear and effective, often less technical, language in order to convey their message to learners. Specific subject-related vocabulary can be explained in a more conversational style or uncomplicated manner than it can be in written format. RAF is often more nuanced than written feedback, with meanings being derived not only from the spoken words but also from the tone of voice, which can be used to convey an overall impression of the feedback. When using RAF, there is a presumption that learners will be able to access it with the appropriate technology or in their e-Education environment. In particular, computing and engineering learners were very interested in this new feedback format. RAF is most appropriate for master’s- and PhD-level

learners who already have basic academic study skills. RAF can be used in a wide range of courses, including giving feedback on reports/paper assignments and PhD paper drafts.

It is possible for supervisors to give richer feedback to learners and save time by using RAF. Providing feedback with a different type of media reinforces their message and gives learners a more personal learning experience. RAF complements more traditional feedback practices. We proposed a process model that highlights best practices and guidelines for supervisors to effectively utilise RAF. Especially, in higher education, supervisors should consider the role of their learners in receiving feedback and how supervisors are encouraged and enabled to maximise the many feedback opportunities available to them in e-Education environments.

We conclude, first, that our observations indicate that learners tend to have positive feelings toward RAF in higher e-Education settings. Second, RAF may have the potential to moderate the effects of cultural dimensions. Third, the minor differences surrounding agreement with particular statements (for example the statements 23 and 24, Appendix 1) perhaps indicate that boundaries separating cultural groups can be unclear (cf. globalization). Fourth, in some cases, cultural differences seem inevitable (for example the statement 27, Appendix 1). We need more accurate data and bigger samples to make claims about the extent of cultural effects within specific cultures.

A systematic approach to culturally adjusting RAF processes would be an interesting and innovative issue for further research, especially focusing on the supervisor’s tone of voice: Which would be pedagogically better―culturally sensitive RAF or culturally neutral RAF―in achieving learning objectives in e-Education environments (Heimbürger & Isomöttönen, 2017) ? Which option is the most efficient for achieving learning objectives? From a technical point of view, it would be interesting to compare the use of several embedded audio files on an assignment versus one audio file, including global, middle, micro and meta levels of RAF.

As more and more higher education activities are performed online, learners have become global and widespread rather than local. The boundaries separating cultural groups are blurred. In effect, 21st-century training and content providers and educational institutions would like to promote a shift towards more culture-neutral e-Education (EADTU, 2017) . This is a challenge because culture not only affects how we behave and think but also how we learn (Henderson, 2006; Olaniran, 2009) . Our further research interest lies in the tension between the two extremes of culture-sensitivity and culture-neutrality. We are interested in moderating cultural effects and in developing a more culture-neutral approach to feedback procedures.

Acknowledgements

The author thanks the Rector’s Office and Education Council of the University of Jyväskylä for supporting our study.

Cite this paper

Heimbürger, A. (2018). Using Recorded Audio Feedback in Cross-Cultural e-Education Environments to Enhance Assessment Practices in a Higher Education. Advances in Applied Sociology, 8, 106-124. https://doi.org/10.4236/aasoci.2018.82007

References

Appendix 1

1) How helpful you think your supervisor’s RAF could be? (Options: very helpful, helpful, somewhat helpful, not helpful)

2) If given the option, which type of supervisor’s feedback would you choose? Please explain your answer shortly. (Options: written only, recorded audio only, combination of written + audio, no preference)

3) In RAF your supervisor is able to use clear and effective, less technical, language in order to convey his/her message. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

4) When specific subject-related vocabulary is used, this can be explained by means of RAF in more conversational style or uncomplicated manner than it would be in written feedback. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

5) RAF is often more nuanced than written feedback. Meaning can be derived from not only the spoken words but also the tone of your supervisor’s voice, which he/she could use to convey an overall impression of the message. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

6) RAF will decrease social distance between the student and the supervisor, and will make the given feedback more personal. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

7) Structure of RAF. Which order is the best? Please, indicate the order with numbers (1 = this issue first, 2 = this issue second, etc.) (Options: paragraph by paragraph comments, general comments concerning the whole document, section by section comments, summary of the next steps, appreciations, critical comments)

8) Which is better? (Options: one audio file indicating clearly the points in the document, for example: page 3, second paragraph, third line or several audio files embedded in the PDF-document in various locations)

9) Please, propose (shortly) a technical solution for realizing RAF in the following two cases: in an e-learning environment, in non e-learning environment

10) What would be the suitable length of the RAF? (Options: 5 minutes, 10 - 15 minutes, 30 minutes)

11) Which would be the three most important issues for you in RAF? Please choose three and mark the order by 1, 2 and 3. (Options: identifying errors, giving praises, correction of errors, explaining misunderstandings, demonstrating correct practices, engaging you to thinking, suggesting further reading, asking you to justify your solution(s) in your assignment, suggesting approaches to future assignments)

12) Which option is the best for the tone of your supervisor’s voice in RAF? (Options: formal (very polite), neutral (no expression of feelings), personal (rich in nuances))

13) RAF will help you to see what you have missed out in your assignment. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

14) RAF will help you to see how you can improve your assignment. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

15) I think that RAF would be easy to access. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

16) I think that RAF could be more helpful than written comments. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

17) In the future, I would be interested in receiving feedback as RAF. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

18) RAF can convey more complex thoughts than written feedback. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

19) The tone of my supervisor’s voice can convey information whether the changes to be done are minor or major. (Options: strongly agree, agree, neutral, disagree, strongly disagree)

20) Which option would be the best for you (please, choose only one)? (Options: The supervisor says in RAF exactly how you should correct your assignment; The supervisor proposes in RAF in more dialogue way how you could improve your assignment)

21) Which option would be the best for you (please, choose only one)? (Options: After you have listened the RAF, you will adjust to supervisor’s comments; After you have listened the RAF, you would like to express your own point of view)

22) Which option would be the best for you (please, choose only one)? (Options: In group RAF, praises will be given to the whole group; In group RAF, praises will be given to the best student)

23) Which option would be the best for you (please, choose only one)? (Option A: In learning activities, the focus is on getting the right answer. Ambiguity is to be avoided. Supervisors are expected to have right answers. Option B: In learning activities, the focus is more open-ended like discussions and project type working. Ambiguity is a natural condition. Supervisors can say, “I don’t know”.)

24) Which option would be the best for you (please, choose only one)? (Option A: In learning activities, there is a focus on logical argumentation to find truth and an insistence on single truths based on logical reasoning. Debate and argumentation are learning activities. Being right is the most important. Options B: In learning activities, there is a focus on achieving practical and socially acceptable outcomes and an acceptance of multiple truths based on experience. Consensus building is a learning activity. Being virtuous is the most important.)

25) Which option would be the best for you (please, choose only one)? (Options A: Learners are expected to be goal-oriented. Knowledge is tied to “cause and effect” explanations. There is a focus on stable knowledge and rules. Options B: There is more willingness to work within situational constraints. Knowledge is tied to explanations of systems and situations. There is a focus on evolving and situational knowledge.)

26) Which option would be the best for you (please, choose only one)? (Options A: Instructional activities start and stop promptly. Meetings outside of class time are limited to strict schedules. There are strict deadlines and consequences for missing them. Options B: Instructional activities are allowed to continue as long as they are useful. Boundaries between class and outside class time are more fluid. Work continues towards improvements with less regard for deadlines.)

27) Which option would be the best for you (please, choose only one)? Options A: Time is to be managed. Learning proceeds along a linear path with clear prerequisites and milestones. Goal setting is essential to learning. Opportunities are not to be wasted. Chances do not present themselves twice. The past is irrelevant. Future goals are important. Option B: One adapts to time. Learning is seen as practice towards slowly increasing perfection. Goals are secondary, one adapts to the situation to draw from it as much as possible. Time exists for observation and reflection, and rushing is counter-productive to achievement. Because time is a series of cycles, opportunities recur. When they do, one may make wiser decisions. The past is influential because cycles repeat. One carries the past forward. Repetition is valuable for learning.)