Montse Álvarez. Fundación CYD/CYD Foundation
Mar 16, 2022 11:00 (CEST)
There can be no doubt that the concepts of accountability and transparency have become highly significant in higher education in recent years. The studies that are needed to achieve the standard required go far beyond the traditional accreditation procedures performed by government or independent agencies, which tend to include a self-assessment report by the institutions or programs themselves as well as an external evaluation report. Hence it is increasingly necessary to carry out tests to examine the results of student learning as part of the process of evaluating higher education programs and provider institutions; in other words, the aim is to assess what knowledge and skills students have acquired, not only generic ones that might be considered common to all academic fields, but also the specific ones related to different areas of study.
In several countries there have been initiatives to develop models for assessing learning outcomes. By means of the AHELO project, the OECD tried to take steps in this direction at an international level since higher education, like so many other aspects of economic and social life, has become more and more globalized, and thus international comparison is essential. The AHELO project was considered to be an equivalent in higher education to the OECD's PISA tests for school-age students, but it failed to come to fruition. Nevertheless, it should be pointed out that efforts have continued to be made for some time to develop similar models in different countries.
AHELO – Assessment of Higher Education Learning Outcomes – was an OECD project that aimed to assess the knowledge and skills of university students in the final year of their undergraduate (and similar) studies. It included both their general skills (such as critical thinking, analytical reasoning, problem solving or communication skills) and abilities specific to their field of study, focusing on the capacity of students to extrapolate what they had learned to its application in new contexts. In addition, it was planned to carry out contextual questionnaires to learn more about the characteristics of the student and their learning environment. The goal was to make it possible to compare these anonymous results internationally at the institutional level and to provide data to governments, universities, employers and students on what graduating students knew and could do compared to international standards.
In the first instance, a feasibility study was carried out between 2010 and 2013, with the participation of 17 countries, international experts, representatives of the higher education sector, and sponsors; civil engineering and economics were the subjects chosen in which the knowledge and skills specific to the field of study were to be assessed. The second step was to carry out the main study, a large-scale implementation of the project, building on the achievements of the feasibility study and improving on the deficiencies found in it, between 2015 and 2020. However, this second step was never taken, and the AHELO project was not developed any further. Among the reasons for the project not prospering economic ones were paramount: the cost of the feasibility study, which, we should remember, was carried out in a period of economic pressure after the impact of the international financial crisis, was substantial; the cost of the main study would have been even higher. Apart from the cost involved, there was criticism of the methodological procedures used, and problems arising from the need to draw comparisons (in higher education there are many different institutions, programs, and curriculums, and even the duration of studies is not homogeneous); mention should also be made of the problems of coordination, governance, and internal management and transparency in the decision-making process; and, above all, the opposition of some of the most prestigious universities; it was not without reason that the associations representing higher education institutions in the United States and Canada sent a letter critical of the project to the OECD.
As has already been mentioned, at the national level there are currently initiatives with this objective in countries such as Australia, Brazil, Mexico, the United Kingdom, and the United States. Among them, Brazil provides a good example of how to assess learning outcomes. This project is being implemented through ENADE – the national examination of student performance (Exame Nacional de Desempenho dos Estudantes, in Portuguese) – which since 2004 has been evaluating the learning outcomes of Brazilian students who are finishing their undergraduate studies. The process consists of two parts, one general and one specific, linked to the area of study, and students have four hours to complete the test. The general part, which is common to all and has no connection with the syllabus, consists of 10 questions, eight multiple choice and two short essays, and counts for 25 % of the final grade. The area-specific part, which consists of 27 multiple choice questions and three short essays, accounts for the remaining 75 % and includes evaluation in basic areas contemplated in the syllabus of the educational field analyzed, ranging from more general topics to more specialized ones insofar as the key areas for undergraduate education are concerned, with questions and problems related to various aspects of the curriculum contents. ENADE is part of the comprehensive Brazilian system for evaluating the quality of higher education known as SINAES (National System for the Evaluation of Higher Education). The fields of study are changed by the organizers every three years, and the test is mandatory for all students in their last year of degrees forming part of the academic areas selected for the year in question.
The OECD-developed evaluation programs that are being applied at an international level, and which have been running for some time, are the PISA and PIAAC programs. PISA – Program for International Student Assessment – is a study that evaluates the performance of 15-year-old students – at the end, therefore, of their compulsory period of secondary education, – in mathematics, science, and reading. This evaluation is carried out through standardized exams that have been carried out every three years in the OECD and associated countries since the year 2000. PISA does not evaluate academic knowledge, per se, but rather the ability of students to understand and solve real-life problems through the application of their knowledge in science, mathematics, and reading; therefore, the tests designed are independent of the school curriculum. To date, more than 90 countries and three million pupils have participated in the different programs carried out.
PIAAC – Program for the International Assessment of Adult Competencies – is a study that assesses the skills of adults aged 16 to 65 in literacy (the ability to understand, evaluate, use, and engage with written texts), numeracy (the ability to access, use, interpret, and communicate mathematical information and ideas), and problem solving in technology-rich environments, including how they use their skills at work and in other contexts such as the home and the community. The assessment is designed to be internationally valid and comparable, as well as applicable to all kinds of cultures. Inasmuch as information on the educational background of the participants is also collected, the results of those with tertiary-level qualifications in the PIAAC tests can be obtained from a wide range of countries; however, they cannot be used to evaluate institutions in the same way as the AHELO system would have done or to examine the results in greater depth according to fields of study. The initial results of PIAAC for the first countries included were published in 2013 (referring to the year 2012) to which the results of other nations were added with data from 2015 and 2017, until a total of some 40
OECD countries and partners was reached. The second phase or cycle of the PIAAC began in 2018 and is scheduled to be completed and published in 2023; data will be collected between 2021 and 2022, with the participation of about thirty countries.
Graphs 1 and 2 show the results of the OECD countries in the PISA and PIAAC tests, compared internationally. Among the 10 with the best PISA scores in each of the three areas studied are Estonia, Canada, Korea, and Poland, while Slovenia, Japan, Finland, the Netherlands, and New Zealand form part of this group in two of the three areas. On the other hand, the countries that are among the ten with the poorest results in the three areas are Colombia, Mexico, Chile, Greece, Israel, and Turkey. In the case of PIAAC, Sweden, Norway, Finland, the Netherlands, the Czech Republic, and Japan stand out for being among the leading countries in the three graphs shown, whereas, at the other end of the scale, Chile, Mexico, Turkey, Greece, Israel, Italy, and Spain might be mentioned.