The U-Multirank indicators are based on a variety of data sources and data collection tools.
Universities that decided to participate in U-Multirank have provided data for the institution as a whole, as well as for the departments offering degree programmes (if any) related to the selected subject areas covered in the 2020, 2021 and 2022 editions of U-Multirank. Both kinds of data were provided through online questionnaires. To ensure comparability of data across institutions, the questionnaires include guidelines and definitions of all data items requested:
In a first phase, participating institutions provide their data using the questionnaires. Data are then intensively checked by the U-Multirank team, applying both automated and manual checks for consistency, plausibility (including checks of outliers) and missing data. Questions and comments related to the data are communicated to the institutions. These are followed up in a second phase of data provision where universities are invited to clarify, correct and add data to the original questionnaire. After the final submission of the questionnaires, data are checked again and any remaining questions are communicated directly (by email) to the universities. Once all data submissions are finalised and the data is regarded as valid and complete, the indicator scores are calculated. Indicator scores follow the definitions provided in U-Multirank’s indicator books: Indicator Book 2022.
One of the purposes of U-Multirank is to help prospective and increasingly mobile students to make an informed choice about a university. For them the assessment of the learning experience by current students of institutions will provide a unique peer perspective. The 2022 edition of U-Multirank includes data drawn from an online survey of around 100,000 students, asking student opinions about various aspects of their degree programme. The student survey was available in a number of languages (incl. English, French, German, Greek, Italian, Polish, Portuguese, Russian, Spanish, Turkish and Ukrainian). Invitations to participate in the survey were sent out by the universities (by email – or post upon request) to a maximum of 500 students per subject area. The indicators taken from the student survey reflect different aspects of the students’ learning experience. They refer to a particular study programme and hence are used only in the subject rankings. If possible, U-Multirank uses data from national surveys like the CHE student survey, the NOKUT studibarometeret or the Netherlands’ Nationale Studenten Enquete (NSE, organized by Studiekeuze123 and commissioned by the Ministry of Education, Culture and Science in the Netherlands).
A number of U-Multirank institutional and field level indicators are based on bibliometric and patent data included in high-quality, comprehensive international databases. This data is produced by the Centre for Science and Technology Studies (CWTS) at Leiden University.
All indicator scores derived from bibliometric analysis are based on information extracted from publications that are indexed in the CWTS-licensed edition of the Web of Science (WoS) database (Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index). The WoS contains approximately 13,000 active information sources, mostly peer-reviewed scholarly journals. The underlying bibliographic information relates to publications classified as ‘research article’ and ‘review article’. The WoS is currently one of the two best sources covering worldwide science across all disciplines. In order to be able to meaningfully calculate some of the bibliometric indicators, we have imposed a threshold on the number of publications per university (50 WoS publications over the period 2017-2020 for the institution as a whole; 20 WoS publications for individual fields of science in the subject rankings).
Read more on Bibliometric Analysis in U-Multirank.
The data underlying the indicator “Publications cited in patents” is collected from the CWTS-licensed edition of the PATSTAT database. Patent publications usually contain references to other patents and sometimes also to other ‘non-patent’ literature sources. A major part of these non-patent references (NPRs) are citations to scholarly publications published in WoS-indexed sources. The patent database used to collect the NPRs from is the autumn 2021 of the EPO Worldwide Patent Statistical Database (PATSTAT).
Other patent-related indicators (Patents granted; Industry Co-patents) are also based on the same PATSTAT database version. Patent indicators are calculated for all U-Multirank universities that have applied for patents in the period 2010-2019. EPO (European Patent Office) and USPTO (United States Patent and Trademark Office) patent grants are extracted, with counts on the level of patent families. A patent family is “a set of patents taken in various countries to protect a single invention”. For the field-based patent indicators, the number of patent families was broken down into corresponding sub-fields based on existing technology classification schemes.
Read more on patent analysis in U-Multirank.
From the beginning it has been an important objective to reduce the burden of data collection by using publicly available data sources. Especially the use of available data for countries with large higher education systems would be a promising option. Starting with national datasets for two higher education systems: the US system and the UK system, we continuously extended the number of countries where we use national data bases by Australia, the Netherlands, Finland, Sweden, Italy and Ontario, Canada. For the 2022 release, the list of pre-filled countries has been extended with France, Brazil and Chile. Data have been extracted from national datasets for a large number of higher education institutions in the US (307 institutions), the UK (165 institutions), Australia (43 institutions), Finland (36 institutions), Italy (92 institutions), the Netherlands (53 institutions), Ontario (20 institutions), Sweden (39 institutions), France (123 institutions), Brazil (77 institutions) and Chile (25 institutions).
In the UK, the data were retrieved from the publications of the Higher Education Statistics Agency (HESA). A conversion table of what HESA data-element was used for what U-Multirank institutional questionnaire item can be downloaded here.
In the US, the data were retrieved from the Integrated Postsecondary Education Data System (IPEDS) database. The conversion table of IPEDS data-elements and UMR data-elements can be downloaded here.
The Australian prefilled data were retrieved from the database available on the website of the Australian Government, Department of Education, Skills and Employment. The conversion table can be found here.
Data on HEIs in Finland were retrieved from the online Vipunen portal (Education Statistics Finland). The conversion table can be found here.
Italian data were retrieved from public datasources of the Ministero dell’Istruzione, dell’Università e della Ricerca (Cerca Università; Portale dei dati dell’istruzione superior). The conversion table can be found here.
In the Netherlands, the data were retrieved from the publicly available tables of DUO, , VSNU, VH and Nuffic. The conversion table can be found here.
The Ontario data originate from the CODUR database of the Council of Ontario Universities. Conversion tables can be downloaded. The conversion table can be found here.
The datasources used for prefilling Swedish HEIs are the publications of Statistics Sweden (SCB) and the Higher Education Agency (UHA). The conversion table can be found here.
French data were drawn from the open datafiles of the Ministrère de l’Enseignement Supérieur et de la Recherche (Opendata). The conversion table can be found here.
The Brazilian pre-filled data were retrieved from the open databases of CAPES and INEP. The conversion table can be found here.
The data on Chile were retrieved from the datafiles of the Subsecretaría de Educaciõn Superior, Servicio de Información de Educación Superior (SIES), at the mifuturo.cl website. The conversion table can be found here.
Based on the data from the prefilling, several ranking and mapping indicators could be calculated. In addition to the questionnaire-based indicators, bibliometric and patent-related indicators were retrieved for all HEIs for which data are in those databases.
Most of the dimensions gain from the prefilling, in addition to the data retrieval from bibliometric and patent databases. However, with the prefilling and data retrieval from external sources, the prefilled HEIs may generate a performance profile that has coverage in all dimensions and is much richer than the profiles of HEIs for which data from external sources only are used.