MBA rankings are a vital component in the Business School community, but new research by the Association of MBAs and Business Graduates Association (AMBA & BGA) exposes a lack of trust in the ability of these rankings to determine the relative quality of MBA programmes.
Most notably, there is a perceived lack of alignment between MBA rankings and the quality of actual MBA delivery.
AMBA & BGA conducted an online survey of 1,291 Business School stakeholders, including senior Business School professionals, MBA students, MBA graduates, and MBA employers between August and October 2018.
It finds that nine in 10 (90%) Business School stakeholders believe that rankings have at least ‘a fair amount of influence’ on the demand for a School’s MBA programme. However, just 11% think MBA rankings reflect the true performance of MBA programmes ‘very well’. In contrast, approximately a third (34%) do not think rankings reflect an MBA’s performance ‘very well’ or ‘at all well’.
Business School professionals, including Deans and MBA Directors, are the least likely group, among the survey’s responding MBA stakeholders, to agree that rankings reflect MBA performance effectively. Just 4% think they do this ‘very well’ and more than half (52%) do not think rankings measure MBA programmes ‘very well’ or ‘at all well’.
The lack of breadth and transparency of criteria employed
Responding to the survey, one Business School decision maker said: ‘Rankings are based on only a few criteria. But why those and not others? Who decides which criteria are the important ones?’
Another Business School leader added: ‘Criteria are not being made explicit and the rationale behind the choice of criteria is never debated. While most rankings try to assess the financial success of graduates, few are really looking into the relevance and the efficacy of the hard and soft skills being taught in the programme.’
Methodological weaknesses in rankings design
A significant number of MBA stakeholders are also unconvinced about the evidence-collection techniques employed by rankings agencies. One Business School leader said: ‘Sometimes, the methodologies of rankings are ambiguous, or the questions are open [to] interpretation, making it difficult to know whether Schools are providing comparable answers.’
Will Dawes, Research and Insight Manager at AMBA & BGA, said: ‘This perceived lack of transparency and resulting ambiguity could be associated with wider cynicism about the accuracy of MBA rankings. Much of the feedback generated in the survey points to low trust in the way responses are collected, with stakeholders suggesting that self-reporting of evidence could potentially lead to distorted results.’
Dawes added: ‘Others highlight methodological issues, such as the quality of questionnaire design, skewed sampling of survey respondents, and inherent biases towards larger programmes, which have a better opportunity to reach minimum response requirements. Other concerns relate to data verification processes, including the implementation of feedback back-checking and the opportunity for Schools to manipulate data (for example, to retrospectively take advantage of currency fluctuations).’
One Business School professional explained: ‘Questionnaires and the response data being provided can be crafted in such a way as to skew the data to the ends of either the surveying organisation or the responding School. A survey may not be asking the right questions to get to the true areas of excellence of a School and Schools may have to manipulate their data to fit the needs of the surveying organisation.’
Dawes said: ‘Another issue raised in the survey is that excessive use of salary data in rankings provides an inadequate picture of the impact of an MBA programme. Specifically, that this measure is unable to reflect what students want from their MBAs, what the MBA programme is seeking to achieve, overall student satisfaction with the programme, and a rounded view of how it impacts on the student.’
One Programme Manager suggested that salary measurements are too short term, adding: ‘Rankings often put too much emphasis and weight on financial gain and often within three months of graduation.’
Other survey participants pointed to students’ objectives, which may not be conducive to earning significantly higher salaries, such as those who become entrepreneurs. Furthermore, salary measurements subsume a number of factors which are intrinsically linked to the personal attributes of the student, rather than the input of the Business School.
Stakeholders also highlight measurement issues with salaries, such as the performance of local economies or living costs, which may distort measurements of the value added from the programme.
One Business School professional described his or her own issues with salaries: ‘The emphasis on criteria relating to income levels pre- and post-MBA disadvantages Schools which, for example, have lots of students moving into self-employment or the third sector. The course might be an excellent personal and business education, but its true performance in relation to AMBA criteria might never be recognised due to the destinations of its graduates.’
A senior Business School professional summarised: ‘The majority of rankings are determined by financial data, and that depends upon the location of the respondents. A great programme in a country that has a low purchasing power parity value is disadvantaged. Also, they don’t recognise programmes that have a focus on areas such as innovation management or entrepreneurship, where managers are neither likely to, nor inclined to, [want] executive positions and thus won’t see the huge salary increases.’
Are rankings out of touch with reality?
Dawes said: ‘The findings from this study suggest that MBA rankings are largely seen to be out of touch with the delivery of MBAs on the ground.
‘One cause might be that the measurements used to assess the quality of MBAs are outdated – with the recent developments in programme design, student compositions and the evolution of what an “MBA” means to students and Schools – and that further consideration needs to be given to rankings modernisation.’