The Businessweek MBA Ranking Controversy
Maria |
October 19, 2021

Ajani Jain, Deputy Dean of Yale School of Management, sparked a controversy with his recent analysis demonstrating how inaccurate and distorted the Bloomberg Businessweek is, in terms of their MBA ranking and methodology, when he attempted to recreate their approach.

In this episode of Business Casual, Maria, John and Caroline discuss the Businessweek MBA rankings and their overall impact. Listen to hear discussion on:

 

  • Why all three of them were already skeptical about the “rankings” (And what exactly John means by that)
  • Why Maria thinks that of all the rankings, Businessweek is always the one with the ‘wild swings’
  • Why Maria praises Jain’s writing on how he approached the matter with Bloomberg in this episode. (It is certainly without a doubt intellectually humorous!)
  • The team debate the question “Do MBA rankings really matter?” Is it more important than a genuine and authentic reputation that data analysis and statistical ranking systems do not obtain?
  • Is ranking manipulation a criminal offense punishable by law?
  • One fact that will enrage the faculty: applicants value rankings three times more than faculty quality when choosing a program. (The statistics gathered from applicants will astound you!)

Episode Transcript

[00:00:07.330] – John

Well, hello, everyone. This is John Byrne with Poets and Quants. Welcome to Business Casual, our weekly podcast. We’re going to talk about a big controversy that’s in the news right now. We broke the story upon Poets and Quants. It’s an analysis by Anjani Jain, the deputy Dean of Yale School of Management, on the latest Businessweek ranking of MBA programs. He pulled apart all the numbers and put them into an optimization model, and Lo and behold, found that he could not replicate their ranking if he used the exact weights on five indexes that they used to measure the quality of an MBA experience. In fact, if he runs the analysis on the weights that they provide and state in their methodology, 79 of the 84 schools that would be ranked would have a different rank, and some of the schools would have dramatically changed rankings. One glaring example, Wharton, which already was ranked 9th in the Businessweek ranking, would actually fall to 28th if the proper weights were accorded based on this analysis by Anjani Jain. Now I should point out that Businessweek is saying that Anjani’s analysis is incorrect. It’s inaccurate. You can’t possibly attempt to replicate the data without having the raw scores as opposed to the scores that they actually published online and in the magazine.

 

[00:01:50.930] – John

Anjani argues that it doesn’t make any difference if the actual scores published in the magazine and online are truly reflective of the raw scores. It doesn’t matter whether you have the raw scores or not. Caroline, what do you make of this?

 

[00:02:06.100] – Caroline

Yeah, I agree that it looks very dodgy indeed, and that Businessweek has not responded very clearly to the challenges, so definitely looks very suspicious. I think they’re trying to make some excuses now and close the discussion, but it definitely looks like something is a mischief, which is very concerning.

 

[00:02:29.450] – John

Yeah, exactly. And now, Maria, I know. Well, all three of us are skeptical of these rankings to begin with. If you’ve listened to our podcast in the past, you know that. And we had a whole podcast not long ago devoted to the Bloomberg Businessweek ranking, where we kind of tore it apart. But, Maria, what’s your take on all this?

 

[00:02:50.420] – Maria

Yeah, I think of all of the rankings, Bloomberg Businessweek has always been the one that kind of has these wild swings back and forth. And I think it’s because my impression is that it’s a very subjective ranking. And so if you organize a bunch of people together and you say, okay, guys, Businessweek is going to be reaching out to you, and it’s really important that you say that our academic experience was the best thing in the world because that’s how they rank, whether or not it’s a good academic program. I don’t think that you should rank a business school based on a popularity contest solely on people’s opinions. Or what if a bunch of graduates are grumpy in a particular year? And I don’t know, I do think that there should be some emotional component. Right. Are you happy you went to the school? Something like that. But I don’t think that so much of it should be so qualitative. So, yeah, I think there’s just something fundamentally wrong with Businessweek, how they’ve been doing this anyway. And so even if they had like, let’s say, let’s say if what Deputy D and Jain is saying is correct, and they got these weird results, like super weird results.

 

[00:04:00.710] – Maria

And they were like, oh, no, this doesn’t look good. So we need to kind of mix things up a little and change it around. Even if that’s true, the fact that they are getting such weird results, that should be a signal to them as an organization. Like, wait a minute, this is weird. One school in Texas, does it really have the best academic experience? I’m sure it’s a great academic school, but isn’t really the best. Maybe we should look inward and with a bit of maturity and say maybe our own ranking isn’t very our own methodology is not that great. I will say, as a side note.

 

[00:04:35.510] – John

In context of what you just said, Marie, if I may, Marie is referring to the fact that under the Businessweek surveys of students, UT Dallas is number one in terms of their learning category, which seems pretty surprising, right?

 

[00:04:54.130] – Maria

I’m sure, I’m sure it’s an amazing school. But come on, the number one learning. Come on. Even when they first get that data back, even before they move to analyze it, and before they move to do all these machinations and this sort of Cirque du Soleil twisting themselves into a pretzel to try to get the to try to claim that their methodology works. Like, maybe internally they should be like, maybe our methodology is kind of lousy because this doesn’t really pass the test. Although I will say, just as a side note for all of this, if I were Businessweek, I would sort of snarkily point out that, hey, Jain, when you redo it, it just so happens at your own school, jumps up four points of four places. And it just so happens that Yale is now again in the top ten, especially because we know, I think we might have mentioned at some point before in one of our conversations that I think Yale in particular has been really trying to game the rankings or to increase the ranking these past several years, anecdotally I’ve seen a much bigger emphasis on GMAT scores at Yale, for example, than I did 15 years ago.

 

[00:06:09.550] – Maria

If I were Businessweek, I would sort of reply with that. But regardless of where you stand on this, this is like, get out the popcorn, read this article, go to Poets and Quants. And right now, because it is such great, it’s such compelling reading, although I don’t remember what his E score is to save my life. But the other thing that’s great is that really just like James writing is so awesome. I love my favorite phrase that he uses it’s a deceitful canard. As a society, we don’t use the term canard enough. And I, for one, applaud Deputy Dean Jain for his wonderful writing, which has just been a delight to read, even if I don’t understand the statistics behind it.

 

[00:06:53.890] – John

Yeah. His response to Businessweek is a really wonderful reading. He calls Businessweek statement disingenuous and nonsensical and yes, a canard, among other things. And then there’s all the analysis, which some of it is above my pay grade in my head. And I never had the benefit, like you two, of taking statistical courses. So while you 2 may know everything about this, including the difference between a normalized score and a Z score, I don’t have a clue.

 

[00:07:33.950] – Maria

Don’t look here.

 

[00:07:36.110] – John

Now, again, I need to point out that I’m the culprit who originally created the MBA ranking back in 1988. But I want to add some context here. The ranking has changed dramatically. Back when I did it, it was a simple customer satisfaction survey that measured the satisfaction of the latest graduating class and the employers who hired them. That’s it simple. And it was cleaned in that way and pure because it was so simple. And I had hired a PhD in statistics who ended up becoming the President of Cooney, the College system in New York, to be my consultant to make sure that we were doing the right things. And I think a lot of these ranking organizations put the fear of God in business schools to report correct data and not to interfere with any surveys of student opinions that they may do. It’s also incumbent on them to have transparency, to operate with full integrity, and to provide the data that would actually allow for more rigorous analysis of the results. And, of course, that’s not happening here. The big question is just an interesting, entertaining fight that we can sit by the sidelines with a bag full of popcorn and enjoy.

 

[00:09:03.350] – John

Or does this fight really matter?

 

[00:09:05.430] – Caroline

Caroline, the rankings do carry rates, right? People pay a lot of attention to them, for better or for worse, from prospective students who are thinking about ways to apply to recruiters, to faculty who are looking at which schools they want to work at. It does have a broad impact. And so they have some responsibility here to respond more clearly to the claims, I think.

 

[00:09:35.190] – John

Yeah, that’s really true. It’s worth pointing out that there’s only one of the five major rankings that actually does audits of school provided information, and that’s the Financial Times. And I believe that they audit programs every three years. And I got to tip my hat off to them for actually going to that extra step to ensure more integrity by the schools, because obviously the schools are under significant pressure to push numbers in their favor, no matter what they are. But to your point, rankings are consequential. They do matter. Two years ago, the association of International and Graduate Admissions Consulting, we call that AJAC. And we’re here as a member. They surveyed MBA applicants, and out of nearly 2000 responses, here’s what they found. They found that nothing mattered more in school selection than rankings and reputation, two factors that, let’s face it, that largely become inseparable. So when you ask the applicants what were the top factors influencing specific choice of schools, 63% of them sided rankings with an equal number of sighting reputation. Now, I’m going to state a fact that it’s going to enraise a lot of the faculty who listen to our podcast.

 

[00:11:02.270] – John

That means, according to this survey of applicants, that applicants value rankings three times more in selecting the program than faculty quality. Faculty quality got 21%. The cost of the program got 23%. GMAT or GRE scores attained, meaning it affects where you apply, 27%. Think about that. 63% said ranking, 63% said reputation. They both go hand in hand. It’s kind of remarkable, frankly. And then when GMAT surveys its folks and they did so in 2021, earlier this year, they found that rankings was a key information resource for researching programs. 45% of domestic MBA applicants said it was 49% of international MBA applicants said it was. So it’s clear that these rankings sway people one way or the other because they do affect your impression of brand. And this is particularly true when you get past Harvest Erin Wharton, and then you start getting into the other schools, which, after all, enroll and get applications from far greater numbers of people than the combined numbers at Harvard, Stanford and Wharton, where I think no matter what their rankings are, it really doesn’t matter because people won’t believe that they’re ranked poorly. No one’s going to believe Wharton’s current rank of nine in Bloomberg Businessweek, and no one would believe the reconstituted ranking, which shows Wharton is number 28.

 

[00:12:45.070] – John

So it doesn’t matter for those schools, but it matters for everybody else pretty much. And then you have this sense that, look, you know, it matters because people cheat. Next month, the former Dean of Temple University’s business school is actually going on trial in federal court for allegedly cooking the books and providing fraudulent data US news that allowed his online MBA program to be ranked number one for four consecutive years. And this is a criminal indictment. It’s the first time the government has actually gone after a school. And it’s been for alleged fraud, which is pretty darn remarkable. And that trial is going to be getting a Philadelphia courtroom, I believe, in the second week of November. Sorry. But Marie, when you chose the school, you weren’t thinking about rankings, were you, because you went to Harvard?

 

[00:13:50.850] – Maria

No. You know, it’s funny. I basically looked at, like, people I admired and where they had gone to school. And then also I looked at people that I didn’t like so much. And a lot of them all randomly, perhaps coincidentally, had all ended up going to another well known school. And so I was like, well, that helps me narrow it down pretty considerably. And yeah, actually, I believe at the time Wharton I mean, Harvard has always been obviously, because it’s Harvard. But Wharton at the time I was living in Asia, and my boss was like, well, Wharton is a better school. Right? And I was like, I don’t know. Wharton is sort of famous worldwide for being like the business school. And so I don’t even know if I looked at the ranking. We’ve talked in the past about why we chose our respective schools, but the people around me were at least my boss at the time was like, oh, I thought Wharton was the better. I’m surprised that you’re taking Harvard INSEAD, but they were just such different programs that I chose the one that I thought was best for me. So I don’t know.

 

[00:14:53.650] – Maria

It’s a shame because I do think that it’s interesting that the three of us have spent so many years in the space, and we’ve seen not just the fluctuations in the rankings, but we’ve seen clients or people that we’ve interviewed or anyone like that who goes to different programs. And many of them have wonderful career outcomes, really, regardless of whether they go to a school that one year is ranked second and the next year is ranked 7th. And so we can see that the rankings themselves, in terms of outcome for your life, don’t really matter a ton. But every year, people coming fresh into this process, it’s a fresh batch of people every single year. And so it’s brand new to them. And so it’s understandable that the rankings is the first place they would look. And I guess it’s understandable that they would place a lot of weight on it. It’s just sort of a shame that we can’t sort of transmit our collective wisdom into their brains and feel like it’s not nice, but it’s not the be all and end all of the experience.

 

[00:15:53.530] – John

That’s really true. And Caroline, obviously you picked INSEAD because you wanted a far more international global experience, and that’s what you got there.

 

[00:16:02.450] – Caroline

Yeah.

 

[00:16:03.550] – John

INSEAD great school was a terrific MBA program. It has been for a very long time. I’m sure rankings didn’t come into the picture for you either.

 

[00:16:14.240] – Caroline

No, they didn’t. I don’t remember consulting the rankings, but I think people it’s something that’s sort of in the background.

 

[00:16:20.490] 

Right.

 

[00:16:20.750] – Caroline

People are aware of them, and I know that it does create a reaction. So I’ve seen that when I was working at it yet that when the score went up in the rankings, stakeholders get very excited. And when the score went down in the rankings, they’d been knocking on the Dean’s door and wanting explanations. So people do take notice. Right. Even people who know these schools incredibly well. Alumni, donors, et cetera. They do keep an eye on these things and they want to understand what’s going on. So even though you may know it’s all very well and appreciate its value, regardless of what particular publication says, people are aware of the impact of the rankings have on other people. So the influence that that has and they care about the sort of public reputation of the schools. And so I do think the rankings have disproportionate influence, unfortunately.

 

[00:17:29.450] – John

Absolutely. Well, no matter how this plays out, meaning, will Businessweek ultimately concede that it made an error or will it never do that? And the argument will just be out there? I think the one good outcome of all this is that it makes us do exactly what we should be doing treating these lists more skeptically, looking at them probably more for entertainment value than for the reality that this school is number three and that school is number five and the other school is number eight. Again, I’ll always say the benefit of a ranking is generally the underlying data that comes with it. And without rankings, I don’t think we would have had the kind of disclosures that we’ve seen in the last quarter of a century where you know what the average GM out of entering classes, you know the average GPA, the number of years of work experience, you know what the outcomes are going to be in terms of average salary, signing bonus, percentage of people who have a job offer at graduation and three months later. These are all stats that often go into rankings and they are public because the rankings exist.

 

[00:18:48.140] – John

And I think that’s a positive thing for people to help make informed decisions about where to go. Meantime, pop your popcorn set in your sofa and read all the stories and Poets and Quants in the back and forth between Bloomberg this week and Anjani Jain and make up your own mind. That’s all makes sense. All right, Maria Caroline, thank you so much. Watch again for your wisdom and your smarts. This is John Byrne with Poets and Quants you’ve been listening to Business Casual.

 

Business School News | General | P&Q Podcast "Business Casual"
The Businessweek MBA Ranking Controversy
Maria |
October 19, 2021

Video transcript, for you skimmers out there: 

I love the fact that they. Report on this metric, right? The salary percentage increase, I think is an incredibly valuable metric because there are so many business schools out there that are great for so many people. And at the end of the day, these programs are in fact able to do what a lot of business school applicants are hoping for.

They are in fact able to provide a real change in the trajectory of someone’s career. They are, in fact, able to help people leapfrog. Into a higher career stratum than they would’ve otherwise been able to be in. So from that perspective, I love the fact that the FT reports on the salary percentage increase.

So valuable. I think it helps, when sometimes I talk to people at the beginning of the business school journey, I will frequently hear something like, well, it’s M seven or bust, you know, it’s Harvard, Stanford, Wharton, or bust.

And I’m often like, look, slow your roll, man. There are so many programs out there that are going to get you. They might not be the first ones that you think [00:01:00] of, but wow, does that even matter? I mean, whew. Look at some of these numbers. $170,000. That is nothing to sneeze at, especially if it’s one and a half times more than what you were making before business school.

I mean, wow. , That is life changing. , And these schools can really change people’s lives. And I think it’s important to have this metric available because I think it helps open people’s eyes. To, To be a little bit more open-minded. , And I think that’s wonderful.

Where my little quibble is. Is that I believe this is an important metric to report upon. However, I do not believe that it is a metric that should have significant amount of weight in the rankings because if we think about what is the purpose of a ranking, it is meant to be some sort of a representation of relative quality.

Now rankings. The entire concept of them is flawed the entire, for me, the entire concept of an ordinal ranking is ridiculous. Like school versus two versus four, versus seven versus six . You know, like, there, there’s sort of [00:02:00] these tiny miniature marginal differences. I think that school rankings should instead be in buckets.

Like, here is the top bucket, and then here is the also very good, but just underneath the top bucket, the next bucket. Um, but no one, no one listens to me. Uh, but so anyway, to the extent that a ranking. Is intended to be some sort of a measure of a program’s quality. I don’t think that this metric is one that should be included in the weighting.

Look, again, . Life-changing levels of improvements in salary. But when I look at, okay, so these were the top five programs by the salary percentage increase, but now when I look at it by the weighted salary, right, the top five US programs, by weighted salary, it’s not entirely accurate to say that.

Well, these programs, you start with people who have lower incoming salaries and they end up in the same place as the other programs. The numbers do not [00:03:00] really, , the numbers would tell a slightly different story. So if you look at the weighted salary a few years out for the top five programs by salary,

we’re talking about a $70,000 a year difference, roughly 240 a year versus 170 a year. That’s about a 40% difference, which I don’t think is a small, you know, if we were talking 5%, even 10%, I’d be like, yeah, 10%, that’s nothing. It’s, you know, nothing but 40% I do think is a pretty, I think it’s a pretty significant difference, uh, that is worth noting.

And so. Your point about like, well, they were letting in the people who were already on a, you know, if you were making, let’s see if we can, if we figure out, okay, so if we take this, these numbers, then we can sort of back into what’s an implied pre MBA salary, you know, that would indicate maybe something in the mid sixties before MBA versus, you know, one 10 something, [00:04:00] 1, 1 10, 1 15, for these other programs.

I get your argument. Your argument is like, look, these people were already clearly high achievers prior to business school, and so, mm-hmm. Is it not true then that the business school, like they would’ve continued to be high achievers And in fact, this is true, some of the most successful, financially successful people I know skipped business school altogether and they didn’t need it.

, However, I think GMAC often does, polls or surveys of MBA graduates, and I think the vast majority of them, at a minimum say that they’re glad that they went to business school, that they do feel that it was worth, their time. So. How much of this is,, nature versus nurture.

We, we will never know. , But I would gently push back on the fact that I, because these numbers essentially to the extent that they’re lower than say these numbers, it effectively penalizes thes e schools in this ranking. And for that reason, I don’t think that it should be part of the ranking because you’re penalizing a school for letting in more successful people.

But there’s a benefit. [00:05:00] To attending. Like, first of all, if you are a more successful person, think of the opportunity cost that you’re giving up. So the fact that these schools are able to lure away people to give up two years of their salary, in order to go to business school in the first place, I think is a pretty good indicator of the desirability or the perceived desirability of those programs.

Also, I do think that there is merit to thinking about like, who are my peers going to be in a business school? and. If a school is attracting people who were more successful prior to business school, I actually think that that is an indicator of the quality of the school, not only because it shows the people that are willing to give up those two years of salary, but also think about who the peer group is once someone is in the school.

Right? That means that if you are attending one of these schools. This percentage isn’t as high, but you’re surrounded by people who, prior to business school, were already achieving on a different level. And also after they graduate, they continue to achieve on a different level. True. The slope is not as sharp.

Right. But the.

[00:06:00] Result is a larger number. So I think that this implies that perhaps at the school itself, you might be surrounded by people who are driven. some people might say more competitive, which might not be everyone’s cup of tea, but people who are more driven and also after they graduate, they continue to be driven.

And so I think that also implies something pretty powerful about the ultimate benefit of the network because business school isn’t just the two years you go there and it’s not just that first job you get out of school or that third job you have five years out of school.

it’s also who’s your network gonna be and, and who are you gonna call 10, 15, 20 years after graduation? To invest in your company or to partner with your company or to start a company with. so I do think that there is value to attending a school and to have your peers during school and after school be people who were, for lack of a better term, high performers.

[00:07:00] I don’t think that this should be punished because I do think that this does yield a better business school. Experience and a better result in the long term. And so my quibble, again, I love this metric. I think this is an amazing metric to provide, but my quibble is that this should not be given honestly, any weight at all, and certainly not the high level of weight that it’s given, because again, you’re punishing the schools that, you know, you’re basically indicating that I, what I would say is an indication of quality.

An indirect indication of quality, but an indication of quality all the same. You’re basically punishing the schools that have sort of higher quality, quote unquote, coming in. And, and that to me is. Counterintuitive and kind of wrong. And so that’s why I continue to think that this should not be, uh, reported upon.

Absolutely. Tell us. It’s important. I think it’s great to know. I love using this information, but I don’t think it should be used in terms of like, let’s figure out which programs are the , [00:08:00] quote unquote highest quality programs. But what do you think? What did I miss? let me know. Thanks.

Maria

New around here? I’m an HBS graduate and a proud member (and former Board Member) of AIGAC. I considered opening a high-end boutique admissions consulting firm, but I wanted to make high-quality admissions advice accessible to all, so I “scaled myself” by creating ApplicantLab. ApplicantLab provides the SAME advice as high-end consultants at a much more affordable price. Read our rave reviews on GMATClub, and check out our free trial (no credit card required) today!