Contextualizing ESL factors with PISA results

Early school leaving (ESL) is an important issue on the educational policy agenda both nationally and internationally. This paper aims to contextualise ESL based on one of the landmark skills assessments, the Organisation for Economic Cooperation and Development (OECD)’s Programme for International Student Assessment (PISA). PISA has become an important component of the strategic framework for European cooperation in education and training (ET, 2020). It represents a prominent information source for the Europe 2020 strategy where it is widely used in fields such as literacy, mathematics, early leavers from education and training, low achievers and ICT skills (European Commission, 2013). Based on large nationally representative samples, the PISA study has been examining the performance of 15-year-olds who are still in education in the areas of reading, mathematical and scientific literacy in over 60 countries every 3 years since the year 2000. The data also include information on students’ attitudes and values, school attendance as well as family backgrounds which enables searching for explanations of the differences in their schooling outcomes.

The aim of this paper is to conduct a literature review on studies that have addressed the issue of ESL by utilising the PISA data and results. Using quality indicators on student achievement together with the indicators on the background characteristics of students and schools that are available in the PISA data, research may have identified (some of) the risk factors of ESL which may help in preventing or reducing ESL.

The results of this paper stem from literature searches performed using diverse search engines and bibliographical databases, library catalogues and websites (e.g. Google, Google Scholar, ResearchGate, NUK Mrežnik with access to ERIC, Sage, Springer, and others). Key words used in the search entailed different combinations of the terms ‘early school leaving/leavers’, ‘dropping out, drop out, dropouts’ together with ‘PISA’ or ‘Programme for International Student Assessment’. Due to language constraints the searches were performed in English. When searching for such studies, the definitions of ESL were not specifically limited. The studies described in the following paragraph might therefore employ different definitions; however, it is considered that these definitions have sufficient elements in common to inform the research on ESL in general. Further, no distinction was made between the terms ‘early school leaving’ and ‘dropping out’ so these terms are used interchangeably in this paper.

The literature search revealed only a few, around 20, peer-reviewed studies directly utilising the PISA data to address ESL. This relatively small number of studies may reflect the fact that it is impossible to directly measure ESL in PISA since the study is cross-sectional and only includes students still in education. Moreover, only a decade’s worth of PISA data is available. Nonetheless, in Australia, Canada and Switzerland the PISA student cohorts were surveyed longitudinally with studies following students’ educational pathways in the years after the PISA assessment. The main characteristic of these studies is that it was possible to identify early school leavers explicitly. Profiles of the early school leavers in comparison with other students were also investigated using the achievement and background data collected in the PISA study. The primary findings about contextualising ESL with the PISA study can therefore be drawn from these studies and we first shortly describe these studies and their most important ESL-related findings.

In Canada, the studies were based on the longitudinal Youth in Transition Survey (YITS) with the first cycle in 2000 in which participants aged 15 also participated in PISA and the same young adults were then interviewed every 2 years until the age of 21. The studies based on YITS provide an overview of the nature of the associations between students’ characteristics at age 15 and their educational outcomes at ages 17, 19 and 21, including for those who left school early. In subsequent cycles, some youth dropped out of school. Bushnik, Barr-Telford and Bussiere (2004) examined characteristics measured at age 15 associated with dropping out of school by the age of 17. Similar investigations of data from the third cycle in which the YITS youth were aged 19 were conducted by Knighton and Bussiere (2006) and data analyses from the fourth YITS cycle were published in the OECD (2010). In Australia, the PISA student cohorts were incorporated in longitudinally designed surveys named the Longitudinal Survey of Australian Youth (LSAY). In the context of ESL, Marks (2007) studied the PISA 2003/LSAY cohort and Mahuteau and Mavromaras (2014) the PISA 2006/LSAY cohort. Further, in Switzerland a panel study of the PISA 2000 cohort of students was followed up by the Transitions from Education to Employment survey (TREE) through seven annual survey panels between 2001 and 2007, and an eighth one in 2010. ESL-related findings based on TREE data were derived by Bertschy, Cattaneo and Wolter (2008) along with Mueller and Wolter (2011).

The key finding in all these studies is that the strongest predictor of ESL is the PISA achievement score, which is even stronger than the socio-economic background of students. For Australia, using multilevel models Marks (2007) showed that the greatest influence on ESL was student performance in the PISA test and that this effect was almost four times stronger than that of socio-economic background. Mahuteau and Mavromaras (2014) confirmed that, in addition to student (disadvantaged) background, scores in the PISA test also predict early dropout well, but they pointed out that low PISA achievers are likely to accumulate other types of disadvantage that together then emphasise the probability of dropping out. Overall, the main message of these findings is that a significant share of the effect of student achievement on leaving school can be separated from the socio-economic and other background student characteristics, indicating that strong competencies seem able to help in overcoming the effects of disadvantages.

Such findings have also emerged in other countries. Bertschy, Cattaneo and Wolter (2008), although their study was limited to Swiss students in vocational programmes, found that a higher score in PISA, after controlling for SES, language spoken at home, migration status and region of residence, significantly predicted the probability of finishing vocational training successfully within 3 years. Other, less successful students changed their initial training place, repeated a year within their training or abandoned it altogether and therefore became vocational ESL students. Mueller and Wolter (2011) also revealed the PISA scores’ significance for future educational success by finding an apprenticeship place for Swiss vocational students. However, they took a different approach to analysing the TREE data: they used background student information from PISA to derive predicted scores and then, based on the residuals of the observed scores from the predicted scores, derived definitions of underachievers, overachievers and achieved-as-expected. Their study investigated whether these student groups differed in their probabilities of successfully starting a certifying apprenticeship after completing school. For PISA-underachievers they found a higher probability of either dropping out, repeating a year, changing training or failing the exam in their vocational training compared to students who had achieved as expected in the PISA test but were otherwise similar to underachievers based on their background characteristics. The authors noted that the PISA test scores and their residuals from the expected scores were virtually the only variables in the study that clearly explained the occurrence of problems during apprenticeship training.

The Canadian studies compared the reading proficiency levels of ESL students when they took the PISA test with the levels achieved by students who successfully continued their education. Bushnik, Barr-Telford and Bussiere (2004) found that youth who had dropped out of school by the age 17 or 19 had PISA scores a whole proficiency level [1] lower in reading literacy than their counterparts who continued education or graduated. Although this result did not control for student background variables, the difference of one proficiency level can be considered comparatively large, and indicates a substantial difference in the nature of reading tasks students can perform. Knighton and Bussiere (2006), after taking the effects of student background into account, further found that the odds of completing high school for a student with PISA 2000 reading scores at Level 2 or below were significantly lower than the odds for a student with scores at Level 3 or above. These results suggest a threshold effect with those at Level 2 and below being at particular risk of not completing school by age 19. However, these relationships are not deterministic; a substantial proportion of students with PISA scores at or below Level 2 in fact successfully graduated, while at the same time a notable proportion of students with PISA scores at or above Level 3 had not graduated from high school by age 19. Similarly, after adjusting for background factors, the OECD (2010) reported that students in the bottom quartile of the PISA reading scores at age 15 were much more likely to drop out of secondary school than those in the top quartile.

The results of the Canadian studies are in line with the results of the mentioned Australian studies by showing that the strong association between reading proficiency and educational attainment still holds after adjusting for background factors, such as socio-economic and immigrant status. In addition, another Canadian study by Murdoch, Kamanzi and Doray (2011) further showed that PISA scores at age 15 together with social factors more strongly impact access to this educational level than later persistence in this education.

Along with the PISA score, studies have also investigated the effect of other factors on ESL. Some effects of the socio-economic background having a strong effect on ESL, but not stronger than PISA achievement, have already been mentioned. Marks (2007) investigated the impact of attitudinal variables on ESL and found that more positive attitudes to school as well as attitudes to teachers reduced the odds of leaving school. In contrast, students’ evaluations of the disciplinary climate in their mathematics classes revealed no impact on school leaving. The study also reported that the inclusion of attitudinal variables had little impact on the effects of other variables. For the school-level measures available in PISA, Marks (2007) found no associations with school leaving in Australia. The author concluded that, consistent with the international literature, this indicates that schools do not have a strong independent influence on school leaving. Yet this is not to say that there are no schools with substantially higher or lower levels of school leaving than expected given their students’ characteristics, but only that there are few of such schools and they do not vary in identifiable, systematic ways from other schools (ibid.).

One other study in Europe addressed the issue of explicitly measured ESL by utilising the PISA results. The importance of the PISA achievement for ESL, more specifically for the consequences of ESL, was found by Alphen (2009). The study analysed EU Statistics on Income and Living Conditions (EU-SILC) data for 2005 to 2007 in examining to what extent compositional variation of early school leavers can account for cross-national variation of their income disadvantage relative to higher educated individuals in 21 EU countries. A country’s mean PISA achievement scores were used as one of the indicators of the average quality of available educational resources, together with educational expenditure. Findings showed that the income disadvantage of early school leavers is smaller in countries where the average quality of available educational resources is higher. This means that a higher PISA mean score in a country was found to be associated with smaller disadvantages of ESL.

The studies described so far based their measure of ESL on explicit variables directly indicating whether a student is an early school leaver or not. Yet there are other studies that employed other variables as an indication of ESL, for example a student’s response as to whether they intend to leave school early obtained from a specifically designed questionnaire. Studies measuring the intention to leave school early were conducted in Ireland (Gillece, Cosgrove, & Sofronuiou, 2010; JCES, 2010), Switzerland (Eicher, Staerkle, & Clemence, 2014), Spain (Enguita, Martinez, & Gomez, 2010) and Italy (Alverini & Lucidi, 2011). Notably, the results of the Swiss study that collected data at different points in time in students’ educational pathways showed that dropout intentions reliably predicted actual dropout 1 year later (Eicher, Staerkle, & Clemence, 2014). Some of these studies included the PISA test score as a predictor of ESL while others only used the PISA student background information, chiefly the PISA socio-economic and cultural index, in investigations of the ESL risk factors. In the first set of studies, the PISA test score again proved to be an important predictor of ESL. The studies and their findings are described in greater detail below.

Both studies for Ireland (Gillece, Cosgrove, & Sofronuiou, 2010; JCES, 2010) used PISA 2006 data and achievement in predictions of ESL, measured by an additional national variable indicating whether a student intended to leave school early without completing Grade 12. The studies aimed to identify the individual- and school-level characteristics for students who intended to leave school early, yet by using different approaches. While the JCES (2010) study took the intention to leave school early as an outcome variable, Gillece, Cosgrove and Sofronuiou (2010) used the intention to leave school early as one of the predictors, among other student background characteristics, of their achievement. The latter study found that intending to leave school early was significantly associated with both high and low PISA achievement, namely students who intended to leave school early were more than twice as likely to be low mathematics achievers than medium achievers (also high achievers). Therefore, the PISA achievement was again shown to be related with ESL.

Investigations of PISA background variables as factors in students’ intentions to leave school early revealed similar results as those studies in which ESL was explicitly measured. The JCES (2010) study found several student-level variables were associated with the intention to leave school early, such as PISA achievement, gender, home educational resources and books in the home, while at the school level only one variable, fee waiver, was found to be a significant characteristic. However, for this variable an interaction with home educational resources was found, suggesting that homes where parents were able to provide higher levels of home educational resources might be operating in a protective manner against student ESL intent in schools where fee waiver rates are high (i.e. with higher concentrations of students from low-income families). This study concluded that, while economic deprivation impacts on ESL, a positive and supportive home educational environment, rather than measures of parental income or education, may be a key factor in protecting against ESL.

From their analysis of PISA 2003 data to investigate the influences of background student characteristics on the process of dropping out of school in Spain, Enguita, Martinez and Gomez (2010) found that Spanish boys have a higher probability of repeating a grade than girls, and that a smaller proportion of boys aspire to obtain a post-compulsory educational degree and are therefore more prone to the risk of ESL. Immigrant students born outside of Spain were found to combine high rates of repetition and particularly low scores in the PISA test and therefore being even more prone to the risk of ESL. At the same time, the children of immigrants, who were born in Spain, revealed a similar ESL-risk level as did working-class Spanish students. The study also found that the risk of dropping out is higher for students from non-nuclear families.

Some other interesting findings have been made about associations of attitudinal factors with the intention to leave school early. An Italian longitudinal study by Alverini and Lucidi (2011) used the PISA socio-economic and cultural index as one of the controls in their path-model investigation of the role of students’ self-determined motivation in reducing the intention to drop out from high school. The results showed that self-determined motivation in students accounted for significant variability in the intention to drop out of high school (i.e. low motivation predicted intention to drop out). Moreover, the intention to drop out seemed to have been more directly affected by self-determined motivation than by school achievement and perceived competence.

In Switzerland, Eicher, Staerkle and Clemence (2014) utilised the TREE study data to investigate associations of attitudinal factors with ESL. More specifically, they investigated longitudinally perceived stress and optimism as predictors of dropout intentions over a period of 4 years while controlling for educational performance in PISA, socio-economic status and immigrant status. The study found that both average levels of stress and optimism as well as annually varying levels of stress and optimism affected dropout intentions. In other words, when controlling for the average level of stress and optimism at the personal level, more stress and less optimism than usual led to higher dropout intentions. The authors concluded that dropout intentions are not stable over the student’s time in education and may thus be influenced by periods of stress experienced by students.

A third set of studies emerged from our literature review of research on ESL within the PISA context. The overriding characteristic of this set of studies is that, instead of employing explicit data on whether a subject is an early school leaver or whether they intend to leave school early, low achievement in the PISA test was used as a proxy measurement for the risk of ESL. More specifically, a student was defined at risk of ESL in these studies if they had achieved a score below Level 2 in the PISA scale for reading (or another domain). Most of such studies found in our literature review were conducted for cohorts of Spanish students participating in PISA. This is not surprising since in Spain school failure has been one of the principal problems of the education system in recent decades (Guio Jaimes & Choi de Mendizabal, 2014). By using several cycles of PISA data, two studies (Guio Jaimes & Choi de Mendizabal, 2014; Guio Jaimes, Choi de Mendizabal, & Escardibul Ferra, 2015) found that personal and most household characteristics with a significant influence on the risk of school failure (i.e. PISA achievement) proved to be particularly stable in this influence over time while most of the school-level characteristics were found not to be stable. This again indicates it is no easy task to reveal the effect of school-level variables on ESL.

In the study by Guio Jaimes, Choi de Mendizabal and Escardibul Ferra (2015) an interesting perspective was introduced, namely that the dynamics of the labour market could impact academic performance and students’ decisions to remain at school. The study established that the youth unemployment rate was positively related with the risk of school failure and early dropout from school. The authors concluded that while the youth unemployment rate (which is highly correlated with the adult unemployment rate) could have a two-way effect on ESL, the first being forcing students to abandon the school system to enter the labour market to help support their families and the second discouraging them to do so due to the lack of labour opportunities for them, the first effect appeared to be stronger.

Two studies by Boada, Herrera, Mas, Minarro, Olivella and Riudor (2011) and by García-Pérez, Hidalgo-Hidalgo and Robles-Zurita (2014) used PISA 2006 and PISA 2009 data, respectively, to analyse characteristics of Spanish students at risk of ESL based on low PISA achievement and investigated the associations between this risk and grade repetition. The first study reported that skills in reading and mathematics significantly affected the likelihood of repetition. The second study stated that grade retention has a negative impact on students’ achievement. The authors concluded that repetition may not be the right policy to help weak students and should be combined with other practices to help in closing learning gaps, e.g. remedial education or early childhood interventions.

NESSE (2010) commissioned an independent expert report on ESL in Europe in which the causes, consequences and possible remedies for ESL were examined. To a small extent the report used PISA analyses, conducted by Willms (2006), on whose basis the report concluded there is an important school composition effect on ESL. School composition effect refers to the ways the characteristics of the student body as a whole, especially its social make-up, affect school processes and influence the achievement of students, even after taking individual students’ socio-economic status into account. The report elaborates that a young person – with the same mix of dis/advantages and the same history of school achievement – will leave one school early but would not leave another school early. Within the context of OECD work, a literature review by Lyche (2010) on international research in the ESL field in OECD countries incorporated the PISA results by investigating factors associated with low achievement. It also concluded that PISA shows a clear advantage when attending schools where students come from a more advantaged socio-economic background. These two studies seem to be in contrast with the previously mentioned findings of Marks (2007) and others that school-level measures showed no association with ESL. However, this contrast may stem from the fact the first studies used an explicit measure of ESL while the above-mentioned studies employed an individual’s low PISA achievement as a proxy for ESL. The second set of studies therefore reveals the well-known finding of the school composition effect on an individual’s achievement but which does not seem to transfer easily to cases where ESL is measured explicitly.

In the paper we have reviewed studies addressing the issue of ESL by utilising PISA data and results. Although the number of such studies we found was not large, a pattern emerges in their findings. The most important finding from our literature review is that students’ achievement shown in the PISA test has a powerful impact on their propensity to leave school early. This was clearly shown by the studies assessing ESL through follow-up longitudinal surveys of the PISA student cohorts. This means that strong competencies are important for students’ further (educational) life. The strength of the achievement as a factor in risk of ESL remained significant even after controlling for socio-economic status, for which associations with school achievement are well established. Further, among the factors analysed, the impact showed to be the strongest, even stronger than the impact of socio-economic status. A significant share of the effect of student achievement on leaving school can therefore be separated from the effect of socio-economic background, indicating that strong competencies can help in overcoming the effects of disadvantages.

Studies have generally found common sets of student-level variables associated with (the risk of) ESL that, in some studies, also proved to be stable in time. Such factors tend to be social factors such as gender, parental education, socio-economic status, immigrant status, previous schooling and, as mentioned, PISA scores. At the same time, scarce findings indicated any school-level impact (e.g. average socio-economic status in the school); in Ireland, JCES (2010) found a fee waiver to be significantly associated with the intention to leave school early while Willms (2006) found that school-compositional effect has an influence on an individual student’s achievement. But when ESL was measured explicitly, Marks (2007) did not find a school effect. This may seem contradictory, but it shows that even though there are schools with substantially higher or lower levels of school leaving than expected given their students’ achievement and other characteristics, these schools do not vary in identifiable, systematic ways from other schools in terms of an observable association with ESL rates. This may actually be seen as showing that there is room for schools to exert a positive effect on ESL. Some studies pointed out that school processes indicate possible remedies in the sense that positive values of student-level attitudinal variables to school may help in reducing the odds of ESL when combined with practices to help close learning gaps early.