Responsible rankings: can measurements be more fruitful for students, Schools and society?

University of Bath School of Management Professor and expert in responsible business, Andrew Crane, on how rankings can effect positive change in the industry, and how to address current criticisms. Interview by Tim Banerjee Dhoul

Recent releases of Business School rankings have been accompanied by media coverage of optouts and boycotts in the context of Covid-19. The omittances have only added to the feeling that rankings are in something of a limbo period amid the clamour for much-mooted and comprehensive redesigns that might allow them to better reflect the current business education landscape. 

That rankings would benefit from change is almost universally recognised. In highlighting many of the problems associated with existing MBA rankings, for example, a 2019 report from AMBA & BGA found – in a survey of 1,328 MBA graduates, students and Business School leaders – that only 11% think rankings reflect the true performance of an MBA ‘very well’. Step forward two years, and the Covid-19 pandemic has provided an impetus for innovation within the business education industry. Might it also present the perfect chance to refresh and revitalise rankings? If so, how can they be made more sustainable for the business world we live in now and the world we will live in in the future?

Andrew Crane is a Professor of Business and Society and Director of the Centre for Business, Organisations and Society in the School of Management at the University of Bath, UK. As an expert in responsible business and the changing role of the corporation in the global economy, Business Impact felt Crane was well-placed to offer a view on the role of rankings and how they might function more responsibly, to encourage positive change in the industry.

What do you think the purpose of (Business School) rankings should be? How does this compare to the real function they currently perform in the sector, as you see it? 

I think Business School rankings should have two main purposes. First and foremost, they should be about giving potential students useful and reliable information that will inform their decision-making, especially about which Schools they should apply to, or enter. Without rankings, prospective students have a dearth of good information on which to base their decisions. 

Second, I think rankings can also be helpful in nudging Business Schools towards whatever we might see as desirable behaviours. A good ranking can be a real driver for change inside Schools. So, as long as the ranking measures things that are important, and measures them in a suitable way, they can have really positive effects on what Business Schools value, what they pay attention to, and ultimately what they do on a day-to-day basis. 

Realistically, one of the main functions that most rankings actually perform is to generate income (whether through sales or advertising revenue) for the media organisations that produce them. It is no surprise, in many ways, that most of the main Business School rankings are produced by media organisations because it is these companies that have recognised how much they have to gain from them. 

This doesn’t have to be a bad thing, but it can have some negative side-effects on the other potential purposes. Most rankings, for example, are not as far as I am aware (with the exception of the Corporate Knights Better World MBA ranking) designed with any real vision of what Business Schools should be teaching, or the role they should be playing in society. If this vision is not built into the ranking, it may still provide useful information for prospective students, but it will not push Schools to behave in any desired way and nor will it consider how Business Schools will actually change in response to the ranking. 

Business Schools are often accused of ‘surrendering’ to rankings. Is this a fair criticism in your opinion? 

I don’t know about surrendering, because I think some Business Schools actively lobby the rankings quite a bit to try and influence what they measure and how. But what I do think is a major problem is that there is real incentive for Business Schools to game the rankings. By that, I mean that they might seek to get as high as possible a score on the ranking while doing as little as possible to actually change the fundamentals of what the ranking is supposed to be assessing. 

One upshot of this is that Schools end up investing too much time, attention and money on their rankings management processes rather than the things that are actually supposed to be important like the quality and relevance of their teaching. Also, the things they do end up changing may be those that are most likely to get a boost in the rankings rather than those that improve the educational experience. This certainly doesn’t happen everywhere but the pressures to do so are likely felt everywhere. This wouldn’t be a problem if the rankings were all designed with a clear vision of what they were trying to achieve in terms of School behaviour, but I fear that these are largely unintended consequences of Schools managing the measures rather than the fundamentals underlying the measures. 

Salary measurements in rankings often attract a lot of attention, with rankings agencies defending the need to demonstrate return on investment (ROI) to paying students. In this context, should Business Schools pay more attention to the cost of business education programmes?

Return on investment is, of course, important. But the idea that the return on an education should be purely, or even primarily, measured in terms of future salary is, in my opinion, deeply flawed. It is even worse than thinking firm performance should only be measured in terms of profits or share price. Education has so many more dimensions than just the direct economic return – even an MBA degree. It is about learning how to think and make decisions in different ways, building up new skillsets and networks, developing in who we are as human beings and professionals, and lots more besides. 

ROI needs to take these things into consideration as well. So, for me, Schools and ranking agencies should be spending more time and attention on how to assess, compare and communicate these other outcomes of a degree programme. Then, once they have a better handle of what the real return on the investment is, they can talk seriously about what the right level of investment (i.e. degree cost) should be. 

Do you think that many Business School practitioners might view rankings as a ‘necessary evil’? Either way, would it be fair to label them as ‘evil’ and/or ‘necessary’? 

Personally, I have something of a love-hate relationship with rankings. In the area of responsible business where I focus, rankings like the now-defunct Beyond Grey Pinstripes ranking or the Corporate Knights ranking have, for all their problems, been really helpful in pushing for change inside the Schools I’ve worked in. 

On the contrary, rankings like the Financial Times one, for me at least, are rather more undesirable because of what I see as a negative effect on how we view Business School education – far too much comes down to final salary. So, yes, I think they are ‘necessary’ because they provide an important service for prospective students. But some are certainly more ‘evil’ than others! 

This year, rankings releases have been accompanied with boycotts/optouts due to problems associated with Covid-19. What value, if any, can rankings offer without being able to score all eligible institutions? 

I’m not too concerned about this, providing they capture a meaningful slice of the population of Schools. It is not like applicants have every single School in their consideration set. Most are selecting from a much smaller subset anyway. So as long as there are enough being ranked for students to be able to compare sufficient Schools that they could feasibly apply and get accepted into, I think the rankings are still going to be valuable for students – which, after all, is their main audience.

Who is to blame for the problems associated with rankings? The agencies who produce them, the Schools that provide them with data, the prospective students who consume them, or the employers who might derive their perceptions of quality from them?  

This is a tough question! The problem is that we are sufficiently far enough down the road of rankings now that we effectively have a whole system of actors that are complicit in keeping them going, and sustaining the problems that are associated with them. Perhaps a better way of thinking about it is to consider who is best placed to change things for the better. And there I think the buck stops with those producing the ratings. If there are problems, they should fix them. 

Of course, Schools, students, and employers can – and absolutely should – play a role in agitating for change, and providing resources and momentum to fuel the change. But it is the agencies that have to change their rankings. Or else, we need different actors to come up with better rankings. 

If you could devise a new system for ranking Business Schools worldwide, what would your top three ranking criteria/factors be and how would you measure them? 

I think there is space for a variety of rankings focused on different aspects of what might be considered important aspirations for Business School education. 

Core to these should be at least some assessment of the relative learning gain of students on the relevant programmes of study – that is, what advance has there been in aspects such as students’ knowledge, skills, work-readiness or personal development. 

There are lots of potential factors and measures that could be used here, all of which I think tell us much more about the return on investment of an MBA programme than final salary. I also think it is crucial that Schools are assessed on how well they prepare students for tackling the big social and environmental challenges that are facing us as a society, and as a business sector. 

Personally, I think that any MBA programme that fails to progress its students’ thinking on how to address climate change, for example, is in dereliction of its duty as an institution of management education.  

What is your impression of the Positive Impact Ratings (PIR), launched in 2020?

Well, first off, it is great to see a new ranking entering this space and with a different set of actors behind it compared to the usual suspects. And I love that it is focused on positive impact and the sustainable development goals. 

Where I have my reservations is on the methodology. Any ranking that is entirely based on students’ responses about what their School is doing is going to have some inherent problems. Most students simply lack enough breadth of experience of other Schools at any point in time to be able to judge effectively the performance of their own School. What is their point of comparison on what good performance looks like? 

Students are a good judge of their own educational attainment, but I’m less convinced they are a good judge of how advanced their School is in making a broader societal impact. Most of what they learn about how good their School is on these aspects will come from a single source … the School itself. Those Schools that talk internally a lot, and convincingly, about how great they are in making an impact will be much more likely to have students that think the School is indeed doing a good job on these criteria. 

I worry that rankings like this will end up assessing how good the internal communications are of the School rather than the underlying fundamentals. Don’t get me wrong, encouraging Schools to improve how they inform students about their societal impact is a good thing in itself. And students are a good source of information on many aspects of School performance that directly affect them. But the approach of PIR gets it wrong in my opinion when it comes to how we best assess and influence Business School behaviour in the context of impact. 

Let’s give students a strong voice in assessing the impact of business education on themselves, but as far as impact on the wider world goes, I would rather see assessments from external stakeholders, and more objective, comparable criteria.

Rankings are usually very popular, so how might the business education industry go about replacing them with something that equally engaged prospective students around the world?  What else might allow students to ascertain quality in a crowded market? 

I don’t think you can replace them. Even if you got rid of the current ones, others would spring up in their place because they are serving a student need. 

They don’t even need Schools to actively participate if they base their metrics on publicly available data. So eliminating them isn’t a plausible alternative, for me. But you can certainly change them or complement them with other types of assessments or ways of demonstrating quality, relevance and impact.  

Much like in every other industry, there has been a growth in the last few years of sites that aggregate students’ reviews and feedback on universities and courses, which I am sure play a growing part in informing students’ decisions. I’m kind of surprised that we have not yet seen a platform of this type successfully emerge to dominate the MBA market. But I’m sure it won’t be too long. 

In addition, I would expect more specialised rankings of business programmes to emerge that focus on specific topics, much like the sustainability ones. So, for example, are we going to see a Business School ranking focused specifically on entrepreneurship, or marketing, or finance? The more this kind of fragmentation occurs, the more opportunities there are for digging deeper and devising better metrics to measure what really matters. 

So, paradoxically, the solution to the problems of our current rankings might be to have more of them, not less. 

Andrew Crane is a Professor of Business and Society and Director of the Centre for Business, Organisations and Society in the School of Management at the University of Bath, UK. He is a leading author, researcher, educator and commentator on responsible business. Follow him at @ethicscrane.

This article is taken from Business Impact’s print magazine (edition: May-July 2021).