Main findings of the project

The main findings of the project were as follows:

1. Portugal´s participation in PISA, right at the first cycle held in 2000, is part of what has been called “normative emulation”, based on the work of Verger (2016) and Addey and Sellar (2017, 2019). The decision, which was not consensual within the government team in the Ministry of Education (ME), was taken in a framework in which the Portuguese government wanted to present itself before the international (and national) community as assigning a political priority to education, recognizing in the OECD a privileged international instance able to produce indicators for external evaluation of the performance of Portuguese students.

2. The creation, within the ME, of the technical-administrative structures responsible for Portugal´s participation in this complex process conducted by the OECD was made from the concern with the elaboration and application of national exams in a more “professionalized” and rigorous way. It was the Gabinete de Avaliação Educacional (GAVE), created in 1997, that was assigned the technical responsibility of organizing the participation of Portuguese students in the 2000, 2003, 2006, 2009 and 2012 cycles. As of 2013, this responsibility was transferred to the Instituto de Avaliação Educativa (IAVE), presented as a public institute of special regime endowed with pedagogical, scientific, administrative and financial autonomy.

3. Portugal shows a consistent rise in data between 2000 and 2018, presenting itself as the only country in the European Union with such performance. It was this consistent rise that led the OECD to consider Portugal a “success case”. A detailed analysis shows that the big jump in results took place in the 2009 and 2012 cycles, stagnating in the last cycle in 2018. These results were subject to detailed analysis, raising some hypotheses (e.g., sample composition; “Ronaldo effect” in the representation of the country), which are still being worked on by the team and will be the subject of a future publication.

4. Portugal has participated in several other large-scale evaluations (TIMSS, PIRLS, ICILS, TALIS), promoted by several international organizations, in particular the IEA and the OECD. In TIMSS it has participated since 1995, in PISA since 2000, in PIRLS since 2011, in ICILS since 2013 and in TALIS since 2018. The data analyzed for the student assessments show that Portugal has been showing positive results in the different assessments, despite fluctuations depending on the cycle and the different literacies (reading, science, and math). In the case of TALIS, analysis shows that the number of positive indicators is higher than the number of negative indicators. It also indicates that their perception of the parameters related to their work in diverse contexts is much lower than the average perception of OECD teachers. The opinions of Portuguese principals are not as “pessimistic” as those of teachers, but they are not positive either. The latter study focuses on school principals’ and teachers’ opinions and working conditions.

5. During the process of analyzing PISA results, given the apparent complexity of some items for 15- year-olds, the team decided to add a research activity to the project: replication of the administration of PISA items (selected from the Science items released in 2015 and the Mathematics items released in 2012), for which it was possible to identify results per student in OECD databases. The Notebook was administered to a convenience sample consisting of students from public and private schools in primary and secondary education (9th to 12th grades) and higher education students from various courses and institutions, private and public, from various regions of the country, in a total of 839 students, aged 15 and older. A detailed analysis of the results shows that most classes have results that do not differ significantly from the PISA results of the Portugal sample on the same items in 2015 and 2012 and that almost all age groups have results that also do not differ significantly from the PISA results of the Portugal sample on the same items in 2015 and 2012. Thus, contrary to the hypothesis, it was concluded that the PISA items seem to have adequate complexity for 15-year-old students and that older students, including in higher education, have similar results as 15-year-old students. That is, the skills and knowledge measured by PISA items appear to be the same in 15-year-olds and older youth and young adults. These tentative findings merit further study.

6. In the project we proceeded to explore, analyze and search for meaning in the media representation of PISA in the Público newspaper – how much and how the reception of PISA is presented in the discourse of news and opinion in a national reference daily newspaper considered. The study was developed in two levels of analysis: the first, an analysis of the surface characteristics of all material with the keyword PISA (180 pieces); the second, a qualitative in-depth analysis of the opinion pieces and news pieces. The study showed (i) a progressive evolution of coverage and emphasis given by the newspaper to PISA, signs of its growing credibility and political importance in society and in the media agenda; (ii) the increasing positive tone of the headlines, signaling the recognition of the favorable evolution of Portuguese results in PISA, especially evident in the news, in contrast with a greater negativity in the chronicles/opinion pieces; (iii) the almost absence of the “public voice”, in particular of the direct protagonists of the schools, signaling the limited openness of the newspaper to a socially plural opinion.

7. In Portugal, international comparative studies feed more discussions in the media and in public opinion in general than in the educational community, namely among teachers. We found very few references in the press of the teacher unions (FENPROF and FNE) and of the mathematics and Portuguese teaching associations, so that we can say that the press of teacher associations and unions is not the most adequate instrument to study the position of these entities regarding PISA.

8. At the level of political discourse, there is an appropriation of PISA results as the fruit of political discourse explicitly assumes the link between policies and PISA results, mainly conceptual in curriculum texts and national exams, as a result of a conscious and intentional political action. This influence has no correspondence in the discourse of the actors – teachers, union associations. The last policy cycle in Portugal (2015-2019) will, above all, reinforce the relationships between knowledge and pedagogy in the curriculum.

In addition, on a more global level, referring to the role that OECD intends to assign to PISA:

1. Despite the effort that OECD has been making in recent years, PISA has methodological weaknesses that limit the validity of many of the data presented and consequent extrapolation in extrapolation in terms of policy conclusions.

2. PISA has become a big science project, embedded in a precise theoretical framework, Knowledge Capital Theory (see papers by Hanushek and Woessmann, 2016, 2019). Arguably within that theoretical framework, it presents itself as entirely out of step with other theoretical frameworks and other perspectives in the educational field. Given the limitations and risks of this kind of approach, which assumes itself as a global governance project led by the OECD, it is a duty of social scientists (and of global citizenship) to seek humanistic alternatives of transnational regulation in education policies.