QS Intelligence Unit

Rankings of anything seem very good at attracting attention, and the simpler they are the more easily and effectively they draw attention. If anyone has ever told a clever joke and then been called upon to explain it you will understand what I am referring to, by the time your audience has understood the joke it has ceased to fulfil its primary purpose – to make people laugh.

There is a great deal of chatter online at the moment – speculation about what newly released rankings might look like, what will be included and what won’t the new THE/Thomson exercise and the CHERPA project through the European Commission are generating particular speculation. The premise on which both of these projects are being discussed is that existing rankings do not fairly measure every aspect of university quality, nor do they recognise the differing nature and structure of different institutions.

Any ranking operated on a global level will be constrained by the quality and quantity of data available and the opinion of its designers and contributors. The worrying trend at the moment is that two underlying assumptions seem to be beginning to resonate throughout this discussion:

1. There is a “perfect solution” – or at least one that will meet with dramatically higher acceptance than those already put forward, and;
2. The stakeholders in rankings are like lemmings and will automatically accept the conclusions of one, or the average of all rankings they consider respectable

The CHE is at the opposite end of the scale to Shanghai and QS methodologies – it gathers masses of data from Germany and surrounding countries but doesn’t actually rank institutions or aggregate indicators – their argument, and perhaps it is a valid one, is that it is not for them to decide what represents quality in the mind of the average stakeholder – particularly students. Fair enough but, broadly speaking, the more proscriptive rankings are not making this assertion either. To my knowledge neither Shanghai Jiao Tong nor QS have ever asserted that their results should be used as the only input to important decisions – the responsibility for such decisions remain the responsibility of the individual making them.

The focus of new developments seems to be on working to the needs and demands of the institutions being evaluated, rather than addressing the needs of the people using them. If such a thing is possible, developing a completely fair and even-handed evaluation, only comparing like against like, is going to become exponentially complex, involving tens, perhaps even hundreds of distinct indicators each engineered in deep technical ways to counter for discipline bias, cultural variety, financial environment, response rate, institution typology, focus, age to a degree that, however transparent the approach is intended to be – its complexity will serve to cloud understanding and the time involved to retrieve and understand results may be off putting.

So the assumptions above seem flawed – it would be irresponsible of any ranking or evaluation to suggest that it is sufficiently complete to be the sole source of data for effective decision making – this increased complexity will promote this illusion rather than allay it and, frankly, the vast majority of people referring to existing rankings are shrewd enough to not take them as more than a single input to their decision making process.

I am, personally, looking forward to seeing what emerges from some of these new projects but in order to achieve some of their bold stated objectives, they are likely to have to sacrifice simplicity, which is not necessarily in the interests of the user.

The THE – QS World University Rankings attract a great deal of interest and scrutiny each year, one piece of frequent feedback is the comparing “apples with oranges” observation. The simple fact is that the London School of Economics bears little resemblance to Harvard University in terms of funding, scale, location, mission, output or virtually any other aspect one may be called upon to consider – so how is it valid to include them both in the same ranking. They do, however, both aim to teach students and produce research and it has always been the assertion of QS and Times Higher Education that this ought to provide a sufficient basis for comparison.

In essence, it is a little like comparing sportspeople from different disciplines in a “World’s greatest sportsperson” or “World’s greatest Olympian” ranking which so frequently emerge. How is it possible to compare a swimmer with a rower with a boxer with a football player? Yet such comparisons have fuelled passionate conversation all over the world. The difference, perhaps, is that in that context those talking are aware of who represents what sport. That is where the classifications come in – they are a component appearing in the tables from 2009 that help the user distinguish the boxers from footballers, so to speak.

The Berlin Principles (a set of recommendations for the delivery of university rankings) assert that any comparative exercise ought to take into account the different typologies of its subject institutions, whilst an aggregate list will continue to be produced it will now feature labels so that institutions (and their stakeholders) of different types can easily understand their performance not only overall but also with respect to institutions of a similar nature.

Based very loosely on the Carnegie Classification of Institutions of Higher Education in the US, but operated on a much simpler basis, these classifications take into account three key aspects of each university to assign their label.

1. Size – based on the (full time equivalent) size of the degree-seeking student body. Where an FTE number is not provided or available, one will be estimated based on common characteristics of other institutions in the country or region in question
2. Subject Range – four categories based on the institution’s provision of programs in the five broad faculty areas used in the university rankings. Due to radically different publication habits and patterns in medicine, an additional category is added based on whether the subject institution has a medical school
3. Research Activity Level – four levels of research activity evaluated based on the number of documents retrievable from Scopus in the five year period preceding the application of the classification. The thresholds required to reach the different levels are different dependent on the institutions pre-classification on aspects 1 and 2.

This will result in each subject institution being grouped under a simple alpha-numeric classification code (i.e. A1 or H3. Table 1 lays out the thresholds for the application of the classifications.

The intention is not to infer a hierarchy – the ranking exists for that purpose – A1 is not a fundamentally preferable classification to G3, but to qualify the subject institutions by broad type with a view to making ranking results more contextually relevant to their increasingly broad audience.

Table 1: Thresholds for application of QS Classifications

[1] Faculty areas are the 5 faculty areas covered by the THE – QS World University Rankings Academic Peer Review: Arts & Humanities; Engineering & Technology; Life Sciences & Medicine; Natural & Physical Sciences; Social Sciences

[2] Research activity levels are defined against thresholds in terms of number of papers identified in Scopus for a 5 year period

Examples
A1 = Large; Fully Comprehensive; Very High Research Activity (e.g. Harvard, Cambridge, NUS)
A2 = Large; Fully Comprehensive; High Research Activity (e.g. Auckland, University College Dublin)
G1 = Medium-sized; Focused; Very High Research Activity (e.g. Tokyo Institute of Technology)
H1 = Medium-sized; Specialist; Very High Research Activity (e.g. London School of Economics)