MBA rankings are a vital component in the Business School community, but new research by the Association of MBAs and Business Graduates Association (AMBA & BGA) exposes a lack of trust in the ability of these rankings to determine the relative quality of MBA programmes.

Most notably, there is a perceived lack of alignment between MBA rankings and the quality of actual MBA delivery.

Key findings

• AMBA & BGA conducted an online survey of 1,291 Business School stakeholders, including senior Business School professionals, MBA students, MBA graduates, and MBA employers between August and October 2018.
• Nine in 10 (90%) Business School stakeholders believe that rankings have ‘at least a fair amount of influence’ on the demand for a School’s MBA programme.
• Approximately one in 10 (11%) think MBA rankings reflect the true performance of MBA programmes ‘very well’.
• The majority (59%) have views on changes they would make to MBA rankings (just 12% of stakeholders do not think any changes are needed while 29% do not know what changes should be made).
• MBA stakeholders are not particularly confident that MBA rankings help students determine the quality of an MBA (just 29% ‘strongly agree’, while 51% ‘tend to agree’ that this is the case).
• MBA stakeholders were asked to state what percentage (or score out of 100) they would give to each of five broad MBA rankings criteria when deciding the overall composition of an MBA ranking. Overall, there is a relatively balanced split between the weight that stakeholders believe should be applied to each of the criteria. The highest mean weighting was given to ‘the quality of management faculty e.g. qualifications, quality of research’ (25%), closely followed by ‘the overall student experience’ (24%). The next highest weighting suggested was for ‘the outcomes of MBA graduates e.g. career progression and salary’ (21%) and ‘the diversity, breadth and quality of the MBA cohort e.g. background of MBAs’ (18%). Meanwhile, ‘alumni engagement e.g. frequency and quality of the network’ is, on average, given a suggested weighting of 13% among this range of criteria.
• For each of these core criteria, participants were also asked about the importance of a range of sub-criteria. Just 34% consider salary to be one of the top two most important factors among graduate outcome measurement criteria (a category which, as indicated above, is deemed worthy of composing a mere 21% of a ranking’s overall score on average).
• For a breakdown of how Business School stakeholders believe rankings should be composed, please see our full report here.

Extended highlights

AMBA & BGA conducted an online survey of 1,291 Business School stakeholders, including senior Business School professionals, MBA students, MBA graduates, and MBA employers between August and October 2018.

It finds that nine in 10 (90%) Business School stakeholders believe that rankings have at least ‘a fair amount of influence’ on the demand for a School’s MBA programme. However, just 11% think MBA rankings reflect the true performance of MBA programmes ‘very well’. In contrast, approximately a third (34%) do not think rankings reflect an MBA’s performance ‘very well’ or ‘at all well’.

Business School professionals, including Deans and MBA Directors, are the least likely group, among the survey’s responding MBA stakeholders, to agree that rankings reflect MBA performance effectively. Just 4% think they do this ‘very well’ and more than half (52%) do not think rankings measure MBA programmes ‘very well’ or ‘at all well’.

The lack of breadth and transparency of criteria employed

Responding to the survey, one Business School decision maker said: ‘Rankings are based on only a few criteria. But why those and not others? Who decides which criteria are the important ones?’

Another Business School leader added: ‘Criteria are not being made explicit and the rationale behind the choice of criteria is never debated. While most rankings try to assess the financial success of graduates, few are really looking into the relevance and the efficacy of the hard and soft skills being taught in the programme.’

Methodological weaknesses in rankings design

A significant number of MBA stakeholders are also unconvinced about the evidence-collection techniques employed by rankings agencies. One Business School leader said: ‘Sometimes, the methodologies of rankings are ambiguous, or the questions are open [to] interpretation, making it difficult to know whether Schools are providing comparable answers.’

Will Dawes, Research and Insight Manager at AMBA & BGA, said: ‘This perceived lack of transparency and resulting ambiguity could be associated with wider cynicism about the accuracy of MBA rankings. Much of the feedback generated in the survey points to low trust in the way responses are collected, with stakeholders suggesting that self-reporting of evidence could potentially lead to distorted results.’

Dawes added: ‘Others highlight methodological issues, such as the quality of questionnaire design, skewed sampling of survey respondents, and inherent biases towards larger programmes, which have a better opportunity to reach minimum response requirements. Other concerns relate to data verification processes, including the implementation of feedback back-checking and the opportunity for Schools to manipulate data (for example, to retrospectively take advantage of currency fluctuations).’

One Business School professional explained: ‘Questionnaires and the response data being provided can be crafted in such a way as to skew the data to the ends of either the surveying organisation or the responding School. A survey may not be asking the right questions to get to the true areas of excellence of a School and Schools may have to manipulate their data to fit the needs of the surveying organisation.’

Salary

Dawes said: ‘Another issue raised in the survey is that excessive use of salary data in rankings provides an inadequate picture of the impact of an MBA programme. Specifically, that this measure is unable to reflect what students want from their MBAs, what the MBA programme is seeking to achieve, overall student satisfaction with the programme, and a rounded view of how it impacts on the student.’

One Programme Manager suggested that salary measurements are too short term, adding: ‘Rankings often put too much emphasis and weight on financial gain and often within three months of graduation.’

Other survey participants pointed to students’ objectives, which may not be conducive to earning significantly higher salaries, such as those who become entrepreneurs. Furthermore, salary measurements subsume a number of factors which are intrinsically linked to the personal attributes of the student, rather than the input of the Business School.

Stakeholders also highlight measurement issues with salaries, such as the performance of local economies or living costs, which may distort measurements of the value added from the programme.

One Business School professional described his or her own issues with salaries: ‘The emphasis on criteria relating to income levels pre- and post-MBA disadvantages Schools which, for example, have lots of students moving into self-employment or the third sector. The course might be an excellent personal and business education, but its true performance in relation to AMBA criteria might never be recognised due to the destinations of its graduates.’

A senior Business School professional summarised: ‘The majority of rankings are determined by financial data, and that depends upon the location of the respondents. A great programme in a country that has a low purchasing power parity value is disadvantaged. Also, they don’t recognise programmes that have a focus on areas such as innovation management or entrepreneurship, where managers are neither likely to, nor inclined to, [want] executive positions and thus won’t see the huge salary increases.’

Are rankings out of touch with reality?

Dawes said: ‘The findings from this study suggest that MBA rankings are largely seen to be out of touch with the delivery of MBAs on the ground.

‘One cause might be that the measurements used to assess the quality of MBAs are outdated – with the recent developments in programme design, student compositions and the evolution of what an “MBA” means to students and Schools – and that further consideration needs to be given to rankings modernisation.’

ENDS