The U-Multirank indicators are based on a variety of data sources and data collection tools.
Universities that decided to participate in U-Multirank have provided data for the institution as a whole, as well as for the departments offering degree programmes (if any) related to the selected subject areas covered in 2019, 2020 and 2021 editions of U-Multirank. Both kinds of data were provided through online questionnaires. To ensure comparability of data across institutions, the questionnaires include guidelines and definitions of all data items requested:
In a first phase, participating institutions provide their data using the questionnaires. Data are then intensively checked by the U-Multirank team, applying both automated and manual checks for consistency, plausibility (including checks of outliers) and missing data. Questions and comments related to the data are communicated to the institutions. These are followed up in a second phase of data provision where universities are invited to clarify, correct and add data to the original questionnaire. After the final submission of the questionnaires, data are checked again and any remaining questions are communicated directly (by email) to the universities. Once all data submissions are finalised and the data is regarded as valid and complete, the indicator scores are calculated. Indicator scores follow the definitions provided in U-Multirank’s indicator books: Indicator Book 2021.
One of the purposes of U-Multirank is to help prospective and increasingly mobile students to make an informed choice about a university. For them the assessment of the learning experience by current students of institutions will provide a unique peer perspective. The 2021 edition of U-Multirank includes data drawn from an online survey of around 100,000 students, asking student opinions about various aspects of their degree programme. The student survey was available in a number of languages (incl. English, French, German, Greek, Italian, Polish, Portuguese, Russian, Spanish, Turkish and Ukrainian). Invitations to participate in the survey were sent out by the universities (by email – or post upon request) to a maximum of 500 students per subject area. The indicators taken from the student survey reflect different aspects of the students’ learning experience. They refer to a particular study programme and hence are used only in the subject rankings. If possible, U-Multirank uses data from national surveys like the CHE student survey, the NOKUT studibarometeret or the Netherlands’ Nationale Studenten Enquete (NSE, organized by Studiekeuze123 and commissioned by the Ministry of Education, Culture and Science in the Netherlands).
A number of U-Multirank institutional and field level indicators are based on bibliometric and patent data included in high-quality, comprehensive international databases. This data is produced by the Centre for Science and Technology Studies (CWTS) at Leiden University.
All indicator scores derived from bibliometric analysis are based on information extracted from publications that are indexed in the CWTS-licensed edition of the Web of Science (WoS) database (Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index). The WoS contains approximately 13,000 active information sources, mostly peer-reviewed scholarly journals. The underlying bibliographic information relates to publications classified as ‘research article’ and ‘review article’. The WoS is currently one of the two best sources covering worldwide science across all disciplines. In order to be able to meaningfully calculate some of the bibliometric indicators, we have imposed a threshold on the number of publications per university (50 WoS publications over the period 2012-2016 for the institution as a whole; 20 WoS publications for individual fields of science in the subject rankings).
Read more on Bibliometric Analysis in U-Multirank.
The data underlying the indicator “Publications cited in patents” is collected from the CWTS-licensed edition of the PATSTAT database. Patent publications usually contain references to other patents and sometimes also to other ‘non-patent’ literature sources. A major part of these non-patent references (NPRs) are citations to scholarly publications published in WoS-indexed sources. The patent database used to collect the NPRs from is the autumn 2018 of the EPO Worldwide Patent Statistical Database (PATSTAT).
Other patent-related indicators (Patents granted; Industry Co-patents) are also based on the same PATSTAT database version. Patent indicators are calculated for all U-Multirank universities that have applied for patents in the period 2004-2013. EPO (European Patent Office) and USPTO (United States Patent and Trademark Office) patent grants are extracted, with counts on the level of patent families. A patent family is “a set of patents taken in various countries to protect a single invention”. For the field-based patent indicators, the number of patent families was broken down into corresponding sub-fields based on existing technology classification schemes.
Read more on patent analysis in U-Multirank.
From the beginning it has been an important objective to reduce the burden of data collection by using publicly available data sources. Especially the use of available data for countries with large higher education systems would be a promising option. Starting with national datasets for two higher education systems: the US system and the UK system, we continuously extended the number of countries where we use national data bases by Australia, the Netherlands, Finland, Sweden, Italy and Ontario, Canada. Data have been extracted from national datasets for a large number of higher education institutions in the US (232 institutions), the UK (165 institutions), Finland (36 institutions), Italy (83 institutions), the Netherlands (50 institutions), Ontario (24 institutions) and Sweden (45 institutions).
In the UK, the data were retrieved from the publications of the Higher Education Statistics Agency (HESA) and from a previous data delivery by HESA. A conversion table of what HESA data-element was used for what U-Multirank institutional questionnaire item can be downloaded here.
In the US, the data were retrieved from the Integrated Postsecondary Education Data System (IPEDS) database. The conversion table of IPEDS data-elements and UMR data-elements can be downloaded here.
In the Netherlands, the data were retrieved from the publicly available tables of DUO, Studiekeuze123, VSNU and VH. The Ontario data originate from the CODUR database of the Council of Ontario Universities. Conversion tables can be downloaded.
Based on the data from the prefilling, several ranking and mapping indicators could be calculated. In addition to the questionnaire-based indicators, bibliometric and patent-related indicators were retrieved for all HEIs for which data are in those databases.
Most of the dimensions gain from the prefilling, in addition to the data retrieval from bibliometric and patent databases. The exception in the knowledge transfer dimension for US HEIs. However, with the prefilling and data retrieval from external sources, the prefilled HEIs may generate a performance profile that has coverage in all dimensions and is much richer than the profiles of HEIs for which data from external sources only are used.