This paper advocates the application of interactive graphics as a qualitative research method for comparative large-scale assessments, known to supplement and extend analytical techniques. Graphics can be easily generated for effective display of information. Interaction is essential for exploring data in a flexible and controlled manner. The national results of the common sample of the large-scale educational assessment studies Internationale Grundschul-Lese-Untersuchung (IGLU)/ Trends in International Mathematics and Science Study (TIMSS) 2011 demonstrated the effectiveness of interactive graphics. The performance profiles of elementary school students in terms of the competency domains of reading, mathematics, and science are analyzed. Furthermore, spineplots are used to investigate the background characteristics of students of different performance types in order to identify possible educational disadvantage. The study aims to simplify data exploration before reporting, and to more effectively present the findings obtained from comparison studies on school achievement.
Large-scale educational assessments have been widely studied. Some of these assessments include the Programme for International Student Assessment (PISA; [
Interactive graphics analyze complex data sets and illustrate the data structure in a flexible and efficient manner. Data can be extracted effectively and effortlessly via interactive graphics [
Ref. [
1) Queries: Interactive graphics retrieve exact or hidden information from a graphic in different ways;
2) Selection: Interactive graphics can effectively compare groups and select data in a variety of ways with a wide range of tools/methods;
3) Highlighting: Through interactive graphics, the user can “link” each selection to all representations of the data and make direct comparisons;
4) Modification of graphic parameters: The parameters of interactive graphics can be varied and adjusted quickly and efficiently.
Based on the reported national (German) scaling results of the common sample of the IGLU/TIMSS 2011 educational surveys, interactive graphics are used as an example in this study to visualize and explore data. In this respect, the appropriate interactive graphics will be used to analyze the main educational aspects or research questions. The present article attempts to make up for the lack of publications on this topic for large-scale assessments, to the best of our knowledge. The software package Mondrian is primarily used to visualize and explore the educational research data, as well as one plot produced in R [
The software program Mondrian helps visualize several data types in various forms. Categorical data can be easily visualized as interactive bar charts, spineplots, and mosaic plots. Continuous data can be displayed as interactive histograms, spinograms, scatterplots, parallel coordinate plots, and box plots. These graphics are described in the following sections. The key benefits of Mondrian are its high level of interactivity and problem- free handling of large data sets, as well as its capability to interact with R to allow for numerical computations from within Mondrian ([
The research organization International Association for the Evaluation of Educational Achievement (IEA) evaluates the reading comprehension, as well as the mathematical and scientific competencies of fourth-grade students once every five and four years, respectively. The PIRLS/IGLU and TIMSS surveys are used for these evaluations. Both the IGLU and TIMSS surveys were simultaneously implemented in Germany for the first time in 2011. The representative German sample comprised 3928 students from 197 elementary schools. Domain- specific student performances were classified based on a scale of five competency levels.
For analyzing particular competencies across domains, the performance test values were mutually scaled with a multidimensional item response model (mixed-coefficients multinomial logit model; for details, see [
However, Germany showed no significant improvements in IGLU 2011, compared with the 2001 and 2006 surveys. Although the social background significantly affected reading comprehension, students with migrant backgrounds did not show a significantly higher reading performance in the 2011 survey [
This paper highlights the advantages of using interactive graphics to analyze comparison studies on school achievement. The aim is to simplify data exploration before the actual reporting, and to graphically present the results.
In addition to the identified performance test values in the three domains, a latent profile analysis (LPA) was conducted. LPA was performed based on the estimated plausible value performance results of the students, with the software Latent GOLD (http://statisticalinnovations.com/products/latentgold.html).
Here, the relationship among the different scales and subscales of IGLU and TIMSS 2011, as well as student groups with similar/different performance or competency profiles, can be determined [
The proportions and performance means of the seven identified performance profiles are presented in
Chief educational aspects or questions are explored and subsequently analyzed with appropriate interactive graphics.
“Educational disadvantage” is the main factor analyzed in comparison studies on school achievement. Although ideally every person has access to the same opportunities towards his or her educational goal [
Type | n | % | Overall reading scale | Overall math scale | Overall science scale | |||
---|---|---|---|---|---|---|---|---|
Ma | SDb | M | SD | M | SD | |||
7 | 165 | 4.3 | 471 | 46 | 482 | 55 | 473 | 50 |
6 | 533 | 13.5 | 410 | 45 | 400 | 58 | 409 | 50 |
5 | 900 | 22.8 | 355 | 43 | 344 | 59 | 350 | 51 |
4 | 1003 | 25.5 | 298 | 44 | 294 | 60 | 298 | 52 |
3 | 753 | 18.9 | 238 | 45 | 243 | 60 | 242 | 49 |
2 | 444 | 11.3 | 164 | 51 | 186 | 57 | 172 | 49 |
1 | 130 | 3.8 | 79 | 60 | 97 | 60 | 83 | 55 |
aMean; bStandard deviation.
students with migrant background, in particular, are disadvantaged in the German education sector according to comparison studies on school achievement such as PIRLS, TIMSS, or PISA [
In light of their relevant cultural and socioeconomic characteristics, as well as their migration background, the present article examines the potential educational disadvantage for students. In this context, the relationships among student performances and several relevant background characteristics are graphically analyzed. More- over, homogeneity within the particular identified performance profiles is analyzed, which depends on the students’ abilities mutually over all three competency domains. On the one hand, the degree of homogeneity within the particular performance profiles can be illustrated and analyzed efficiently through interactive graphics. On the other hand, students with relative performance strengths or weaknesses can be easily evaluated via linked graphics. This easy reckoning can, for instance, identify students with individual learning disabilities or excellent performances in only one domain [
The central educational aspects and research questions are summarized in
Considerable data on background characteristics are available, in addition to the performance test values assessed in the IGLU/TIMSS 2011 surveys. These comprise, inter alia, cultural and social background characteristics, as well as subject-specific attitudes and self-concepts.
This section focuses on the degree of difference among the seven identified performance types, relative to certain characteristics such as gender or cultural and social properties. In terms of their relevant background information, the potential educational disadvantage for students, in particular, can be graphically explored [
In all comparison studies on school achievement, carried out so far in Germany, it was stated that a part of the students does not master basic skills in reading, writing, and arithmetic. […] Evaluations of the […] PISA survey and the […] IGLU survey from the year 2006 clarify the persistent link between education and educational opportunities, in which particularly children with migration background are disadvantaged ([
In an ideal education system, every person has the same opportunities to achieve his or her educational goal. However, in reality, some are disadvantaged in this respect, in particular, students with lesser personal, social, financial, or cultural resources (see also [
This section aims to graphically determine the potential educational disadvantage for students based on their relevant background characteristics. The spineplot, produced easily in Mondrian, can be used to this end.
Educational research question | Interactive technique |
---|---|
Characterization of performance types and identification of educational disadvantage | |
Can educational disadvantage for students be explored using relevant background characteristics? | Spineplot |
Is there a relationship between missing values in the background variables and performance profiles? | Missing value plot, spineplot |
Homogeneity | |
Can homogeneous groups of students be identified with similar performance patterns within the performance profiles? | Parallel coordinate plot |
Misallocations | |
Can misallocations to the performance profiles be identified? | Parallel box plot |
Relative performance strengths and weaknesses | |
Can relative performance strengths and weaknesses be identified for individual students? | Histograms, parallel coordinate plot |
Comparison of domains and performance profiles | |
Can differences between the domains and performance profiles be identified regarding relevant background characteristics? | Trellis plot |
Spineplots are similar to conventional bar charts. However, the conditional probabilities of the analysis can be compared in the former, as the length of the bars is uniformly scaled [
The graphical results of the descriptive relationships among student performances and selected background characteristics are summarized in
Risk of poverty of family: The educational disadvantage of students at a risk of poverty can clearly be explored with the variable risk of poverty of family. In particular, the estimated probability P (no risk of poverty | performance profile Xi) and the estimate P (risk of poverty | performance profile Xi) show a consistently increasing relationship on comparison. Families of students with higher performance profiles are at a lesser or no risk of poverty (
Migration background of parents: Similar patterns of educational disadvantage were also identified based on the variable migration background of parents. In particular, both parents of students with a higher performance profile were significantly more likely to have not been born abroad (
The graphical results for further background variables can be found in Appendix A.1 (see
Although not considered previously, missing values within background variables have been found to be very important in properly evaluating large-scale assessments such as IGLU, TIMSS, or PISA, as they may generally lead to distorted results. Presently, several statistical methods are available to overcome this limitation. The multiple imputation of missing values is most frequently used in educational sciences (e.g., [
In this respect, the structure of the missing values and their proportions within particular variables must be studied. The missing value plot can be produced with the software Mondrian for this purpose. The key advantage of the missing value plot is its ease of use and interpretation, which is addressed in this study. Simple graphical representations of the proportions of missing values within background variables of interest are presented in
The various factors giving rise to missing values are not discussed in this paper. Instead, the present paper aims to identify the potential response tendencies for selected background variables, especially in terms of classifying students into the specified performance profiles. In this respect, spineplots were used to explore the probabilities of a missing value appearing under a particular performance profile, that is, P (missing value [in a variable] | performance profile Xi). The corresponding graphical explorations for the background variables risk of poverty of family and migration background of parents are reported in
Comparable tendencies with respect to the response behaviors of the students can be identified. Students with lower performance profiles are more likely to have missing values within the two background characteristics. For example, P (missing value | performance profile 1) is greater than P (missing value | performance profile 7). The probability of occurrence of a missing value steadily decreases with higher performance profiles.
Similar results were observed with other background variables as well. Therefore, missing values do not seem to be random in this context. Further, the obtained information can be useful in analytical modeling approaches to dealing with missing values, for example.
Via LPA, students with similar abilities in the IGLU/TIMSS 2011 surveys were clustered and classified into highly homogeneous groups (e.g., [
The degree of homogeneity within profiles can be analyzed in various ways with the help of interactive graphics. The parallel coordinate plot [
The performance test values of the students over all three domains with respect to the selected performance profiles 1, 3, 5, and 7 (left: reading dimension; center: mathematics dimension; right: science dimension) are presented in
Performance profile 1 is particularly striking. Some cases that are not homogeneous within that profile can be graphically detected. An arrow-marked extreme case in
Misallocations within the performance profiles must be identified to maintain the high degree of homogeneity and to counteract a heterogeneous pattern within particular profiles. The parallel box plot can be used for this purpose (e.g., [
The parallel box plots over the three competency domains for the corresponding profiles 1, 3, 5, and 7 are presented in
Interestingly, homogeneity within profiles may be distorted by the presence of students with relative performance strengths or weaknesses, such as students with learning difficulties or excellent performances in merely one domain (e.g., see [
their specific domains (e.g., see [
The relative performance strength for the reading domain is graphically illustrated using the software Mondrian (see
This is based on the histograms of the performance test values on the one hand and the parallel coordinate plot of the test values over the three domains on the other [
As presented, the graphical procedure identifying students with relative performance strengths or weaknesses may facilitate subsequent and more advanced analyses. For instance, the possible causes for the detected relative performance strengths and weaknesses can be examined in depth with additional linked graphics for interesting background characteristics.
The reading, mathematics, and science domains and the identified performance profiles were compared, con- sidering the relevant background characteristics.
The so-called trellis plot can efficiently display this comparison graphically, which can be produced by the R package lattice [
In the example of this paper (see
Students exposed to the risk of poverty were found to be explicitly disadvantaged educationally with respect to the background variable risk of poverty of family, as derived from the graphic (cf. also Section 4.1). Educational disadvantage can be identified within all three domains, as well as over most of the performance profiles, with few exceptions. In particular, students not exposed to the risk of poverty achieve comparatively higher performance test values than those who are. However, exceptions were students exposed to the risk of poverty within lower performance profiles. Some of those students exposed to the risk of poverty with lower profiles had higher test values in the reading and mathematics domains. This may be partly explained by the comparatively higher proportions of students exposed to the risk found in the lower performance profiles. However, students within the upper performance profiles who are not exposed to the risk of poverty achieved higher performance test values.
This paper presents the use of interactive graphics to empirically analyze the scaling results obtained from large-scale educational assessment studies. The efficiency of interactive graphics has been demonstrated using empirical data from the IGLU/TIMSS 2011 surveys. It is an easy, intuitive, and effective method of investigating substantial educational questions or hypotheses.
It is important to note that interactive graphics do not necessarily replace analytical scaling procedures such as the item response theory. Graphical and numerical/analytical approaches are complementary, rather than opposing. The information obtained from visual data exploration may serve as a reference for assessing the plausibility of results derived from inferential statistical methods more qualitatively. In this sense, visualization cannot be considered in the same vein as statistical confirmatory procedures. Nevertheless, and probably because of this, interactive graphics enable one to “recognize or see” relatively complex questions effectively.
This method holds future research prospects. Interactive graphics can be very useful in large-scale assessments, where large quantities of data are available and collected over several survey cycles. For example, data can be analyzed more completely with a combination of the item response theory and graphics through mixed- methods strategies. Qualitative and quantitative evidence must be combined systematically and supplementary to each other (e.g., [
Classification may also be addressed in future research. In this regard, several available methods of classification have not yet been applied in large-scale assessments, whereat the complementary use of interactive graphics can prove valuable. From an educational perspective, students with relative performance strengths or weaknesses must be systematically identified with specifically designed procedures for determining these “rare or extreme cases.” In addition, a more fine-grained subdivision or granularity of the competency domains may be necessary for a more accurate search of peculiar performance patterns. In this respect, the comprehension processes and cognitive demands of the students within the individual domains could also be considered during the analyses. These investigations can certainly benefit from the use of interactive statistical graphics.
In conclusion, graphics have the following features: they are generated rapidly, intuitive and relatively simple to understand, and capable of developing their own dynamics during data analysis, especially due to their non- normative/descriptive nature. The effectiveness of graphics is well known. Results are often presented or expressed visually. It follows then that graphics can be systematically and profitably integrated into educational science and research, in addition to complementing previous numerical methods. It is hoped that the present paper can contribute to a research basis in this direction.
AliÜnlü,BernhardGschrey, (2015) Interactive Graphics for Presentation and Exploration of Student Performance Profiles in IGLU/TIMSS 2011 Educational Surveys. Open Journal of Social Sciences,03,122-136. doi: 10.4236/jss.2015.39018
In
The homogeneity and misallocation with respect to the seven identified performance profiles are presented in