Debunking the MBTI Debunkers

(Content from two posts by reckful.)

It seems like the MBTI "debunkings" have been coming thicker and faster lately, but their quality certainly isn't improving — which is hardly surprising, given the extent to which each one seems to be based largely on a quick review of previous "debunkings," rather than on the authors actually doing much serious homework.

I'm going to take more time than the latest debunker really deserves to address some of the points in the article by Joseph Stromberg (a dude who "writes about science" at the Vox website) mentioned in the OP, partly because they're mostly points commonly found in these kinds of articles, so addressing this one also addresses several previous articles, as well as (I assume, alas) several more that are still to come.

The Big Five is science and the MBTI is astrology

I have more to say about the scientific status of the MBTI below, but wanted to begin by noting that, like most MBTI debunkings, this one points approvingly at the Big Five and characterizes it as a very different kind of animal. But McCrae and Costa — the leading Big Five psychologists (and creators of the NEO-PI-R test) — long ago acknowledged (1) that the MBTI (and this was an older version than the current one) basically passed muster in the validity and reliability departments, (2) that the MBTI was effectively tapping into four of the Big Five dimensions, and (3) that the Big Five and the MBTI might each have things to learn from the other.

Discrete, bimodal types

Originally Posted by Stromberg

The test claims that, based on 93 questions, it can group all the people of the world into 16 different discrete "types." ...

With most traits, humans fall on different points along a spectrum. If you ask people whether they prefer to think or feel, or whether they prefer to judge or perceive, the majority will tell you a little of both. ...

But the test is built entirely around the basis that people are all one or the other. ...

Actual data tells psychologists that these traits do not have a bimodal distribution. Tracking a group of people's interactions with others, for instance, shows that as Jung noted, there aren't really pure extroverts and introverts, but mostly people who fall somewhere in between.

Pew! Pew! Pew! And another straw man crumples to the ground...

The notion that the MBTI claims to assign people to "pure" all-or-nothing categories is probably the silliest of the memes that regularly recur in MBTI debunkings, and it has the dual charm of being both an inaccurate characterization of the MBTI and — in its misplaced emphasis on the shape of the distribution curve — a red herring.

Nobody knows for sure at this point but, as I understand it, the existing studies suggest that it's likely that most or all of the MBTI dimensions — like the four Big Five dimensions they basically correspond with — exhibit something like a normal distribution, with substantially more people near (or in) the middle than near the extremes. For what it's worth, Jung thought more people were essentially in the middle on E/I than were significantly extraverted or introverted, and Myers allowed for the possibility of middleness on all four dimensions — so the in-the-middle possibility really goes all the way back to the MBTI's roots.

Myers believed that it might turn out that one or more of the dichotomies was truly bimodal to one degree or another — with, in effect, a more or less empty (if narrow) zone in the exact middle of the spectrum. But she never asserted that that theoretical possibility had been factually established by any respectable body of evidence, and the 1985 MBTI Manual (which she co-authored) stressed that the evidence for bimodality was sketchy at best. And since then, as I've said, quite a lot of evidence has accumulated that seems to suggest that most or all of the MBTI dimensions exhibit something more like a normal distribution.

In at least one of the early versions of the MBTI, it was possible to get an "x" on any dimension. The current version assigns people a (tentative) type on each dimension, but that's a very different thing from saying that it isn't possible for someone not to have a preference — and the MBTI Manual specifically notes that someone with a score near the middle is someone who has essentially "split the vote" rather than offered much evidence of a preference.

The "Step II" version of the MBTI includes five "facets" for each dimension — just as the NEO-PI-R has six facets for each Big Five dimension — and allows for the possibility of being, for example, on the T side of three of the facets and the F side of the other two.

More importantly, I'd say, there was really no doubt in either Jung's or Myers' minds that people on either side of the dimensions fell along a notably wide spectrum from mild to strong preferences. So, regardless of where anybody wants to come down on the "exact middle" possibility, if they take the position that, e.g., all introverts are equally introverted, their perspective is way out of line with Jung, Myers and every respectable MBTI source I've ever encountered.

As a final note: At this point nobody really knows how close to the middle how many people are on the MBTI (and Big Five) dimensions, because the current state of both the MBTI and Big Five is such that it really isn't possible to determine exactly where anybody falls along whatever the real, underlying (and substantially genetic) spectrums may be. So it seems to me that anybody who thinks that the existing data on either the Big Five or MBTI has clearly established the shape of the distribution curves is very much overestimating the ability of the existing tests to accurately quantify strengths of preferences.

But the main point to keep in mind is that, at the end of the day, the worth of the MBTI and Big Five is mostly going to hinge on how good a job those typologies do in nailing down what personality-related characteristics tend to be associated with the corresponding preferences, and not on how many people turn out to be at any particular point on any of the relevant spectrums. And in any case, the MBTI certainly doesn't stand or fall depending on whether any of its dimensions exhibit a "bimodal" distribution.

The MBTI simply implements Jung's types

Originally Posted by Stromberg

The test was developed in the 1940s based off the untested theories of an outdated analytical psychologist named Carl Jung, and is now thoroughly disregarded by the psychology community. ...

It copied Jung's types, but slightly altered the terminology, and modified it so that a person was assigned one possibility or the other in all four categories, based on their answers to a series of two-choice questions. ...

If there were good empirical reasons for these strange binary choices that don't seem to describe the reality we know, we might have reason to seriously consider them. But the fact is that they come from the now-disregarded theories of a early 20th century thinker who believed in things like ESP and the collective unconscious.

Jung was a believer in the scientific approach, and Isabel Myers took Psychological Types and devoted a substantial chunk of her life to putting its typological concepts to the test in accordance with the psychometric standards applicable to the science of personality. Myers adjusted Jung's categories and concepts so that they better fit the data she'd gathered from thousands of subjects, and by the end of the 1950s (as McCrae and Costa have acknowledged), she had a typology (and an instrument) that was respectably tapping into four of the Big Five personality dimensions — long before there really was a Big Five. And twin studies have since shown that identical twins raised in separate households are substantially more likely to match on those dimensions than genetically unrelated pairs, which is further (strong) confirmation that the MBTI dichotomies correspond to real, relatively hard-wired underlying dimensions of personality. They're a long way from being simply theoretical — or pseudoscientific — categories with no respectable evidence behind them.

Again, McCrae and Costa are the leading Big Five psychologists, and they've studied both Jung and the MBTI. In the same article I linked to at the top of this post, they noted — correctly — that Jung's typology erred in lumping various psychological characteristics together that decades of studies have shown are not significantly correlated. By contrast, after Myers was finished adjusting Jung's system to fit the data, she had a modified version whose dichotomies passed muster by the relevant scientific standards. As McCrae and Costa explain:

Originally Posted by McCrae & Costa

Jung's descriptions of what might be considered superficial but objectively observable characteristics often include traits that do not empirically covary. Jung described extraverts as "open, sociable, jovial, or at least friendly and approachable characters," but also as morally conventional and tough-minded in James's sense. Decades of research on the dimension of extraversion show that these attributes simply do not cohere in a single factor. ...

Faced with these difficulties, Myers and Briggs created an instrument by elaborating on the most easily assessed and distinctive traits suggested by Jung's writings and their own observations of individuals they considered exemplars of different types and by relying heavily on traditional psychometric procedures (principally item-scale correlations). Their work produced a set of internally consistent and relatively uncorrelated indices.

Jung included what's arguably the lion's share of the modern conception of S/N (the concrete/abstract duality) in his very broad notion of what E/I involved. But Myers discovered that there are abstract extraverts (ENs) and concrete introverts (ISs), and that there's no significant correlation between Myers' (statistically supportable) versions of E/I and S/N. Jung said extraverts tend to subscribe to the mainstream cultural views of their time, while introverts tend to reject mainstream values in favor of their own individualistic choices. But Myers discovered that a typical ISTJ is significantly more likely to be a traditionalist than a typical (more independent-minded) ENTP. Jung said an extravert likes change and "discovers himself in the fluctuating and changeable," while an introvert resists change and identifies with the "changeless and eternal." But Myers discovered that it was the S/N and J/P dimensions that primarily influenced someone's attitude toward change, rather than whether they were introverted or extraverted.

And so on. The appropriate way to view the Myers-Briggs typology is not as some kind of simplified (and more "testable") implementation of Jung's original typology. Instead, it's fairer to say that the Myers-Briggs typology is basically where Jung's typology ended up after it was very substantially modified — not to mention expanded — to fit the evidence.

Reliability

Originally Posted by Stromberg

We could accept the fact that the Myers-Briggs is limited in defining people in binary categories, but still theoretically get some value out of it because it accurately indicates which pole of any category we're closest to.

But that idea is tough to swallow given the fact that the test is notoriously inconsistent. Research has found that as much as 50 percent of people arrive at a different result the second time they take a test, even if it's just five weeks later.

That's because these traits aren't the ones that are consistently different among people. Most of us vary in these traits over time — depending on our mood when we take the test, for instance, we may or may not think that we sympathize with people. The idea that the Big Five is substantially superior to the MBTI in the test/retest reliability department is another canard that periodically pops up in these kinds of articles. And claims to that effect are often accompanied by statistics that confuse retest rates on single dimensions with retest rates for a complete four-letter type.

I once corrected a forum poster who'd noted that the MBTI "has a test-retest rate of some 60%, meaning two out of every five people get different results when retaking the test," while the NEO-PI-R's "levels of consistency are incredibly high (N= .92, E= .89, O= .87, A= .86, C= .90)." In my reply, I explained:

Originally Posted by reckful

That 60% MBTI statistic relates to a retest standard that says you got a different result if any one of the four dimensions is different. That corresponds to an average test-retest rate of 88% for the individual dimensions.

If you apply the same test-retest standard to those Big Five statistics you gave us, you get .92 * .89 * .87 * .86 * .90 = a 55% test-retest rate (or 60% if you leave out Neuroticism).

It's probably also worth noting that if you assume (as previously discussed) that most or all of the MBTI and Big Five dimensions exhibit something like a normal distribution, and if you assume (accordingly) that a large portion of the population is in or near the middle on at least one dimension, and if you add to that the many potential sources of error in self-assessment personality tests — from the fact that personality type is a relatively young science and psychologists are quite a long ways from nailing down exactly what the temperament dimensions consist of, to flaws in particular tests (including items that tap into more than one dimension), to multiple kinds of misunderstanding and other human error on the part of the individuals taking the test — it would strain credibility if the test-retest statistics for any personality typology didn't indicate a significant percentage of cases where at least one of the dimensions came out with a different preference on retesting, and one letter change is all it takes to constitute an MBTI retest "failure."

As a final note, it should also be kept in mind that a typical MBTI test-taker is someone with little or no familiarity with the typology who simply takes the MBTI test along with a group of fellow employees or students. It's reasonable to assume that, to the extent that a person actually has four reasonably-well-defined preferences, they're likely to come up with a result that's considerably more accurate if, rather than just accepting the test result, they spend some time reading about the preferences and the types — which is something the MBTI Manual (among other sources) has always encouraged people to do.

Myers didn't have a psychology degree!

Originally Posted by Stromberg Jung's principles were later adapted into a test by Katherine Briggs and her daughter Isabel Briggs Myers, a pair of Americans who had no formal training in psychology. To learn the techniques of test-making and statistical analysis, Briggs worked with Edward Hay, an HR manager for a Philadelphia bank.

No "formal training" in psychology! Oh noes!

Isabel Myers may not have been as smart as Jung, but she was a very intelligent woman — she graduated first in her class at Swarthmore — who understood that, in order to create a personality assessment instrument that passed muster by the relevant scientific standards, she needed to educate herself on statistics and psychometrics. And she did. And if Mr. Stromberg thinks that the fact that Myers' education in that area happened outside of a "formal" university program means she didn't really know what she was doing, I'd suggest that Mr. Stromberg should think again.

I'd certainly expect that, all other things being equal, a smart person with a degree in psychology would have been in a better position to turn Psychological Types into a scientifically-respectable typology than a smart person with "no formal training in psychology." But, as it turns out, Briggs and Myers were the smart people who did it, and the Myers-Briggs typology deserves — needless to say, I would hope — to be judged on its merits, rather than on the basis of how much of its creators' education happened within the hallowed halls of academia.

Real psychologists reject the MBTI

Originally Posted by Stromberg

Search for any prominent psychology journal for analysis of personality tests, and you'll find mentions of several different systems that have been developed in the decades since the test was introduced, but not the Myers-Briggs itself. Apart from a few analyses finding it to be flawed, virtually no major psychology journals have published research on the test — almost all of it comes in dubious outlets like The Journal of Psychological Type, which were specifically created for this type of research. ...

Apart from the introversion/extroversion aspect of the Myers-Briggs, the newer, empirically driven tests focus on entirely different categories. The Five Factor model measures people's openness, conscientiousness, extraversion, agreeableness, and neuroticism — factors that do differ widely among people, data has told us. And there's some evidence that this scheme have some predictive power in determining people's ability to be successful at various jobs and in other situations. ...

It's 2014. Thousands of professional psychologists have evaluated the century-old Myers-Briggs, found it to be inaccurate and arbitrary, and devised better systems for evaluating personality. Let's stop using this outdated measure — which has about as much scientific validity as your astrological sign — and move on to something else.

Stromberg packs a lot of misinformation into the closing paragraphs of his article. He says that, except for E/I, the Big Five "focuses on entirely different categories" — and I've already pointed out that the leading Big Five psychologists (and authors of the NEO-PI-R) have come to the opposite conclusion.

He says that, "apart from a few analyses finding it to be flawed, virtually no major psychology journals have published research on the test — almost all of it comes in dubious outlets like The Journal of Psychological Type, which were specifically created for this type of research." But, on the contrary, and as further described in the next linked post, professional psychologists have been publishing studies based on the MBTI in independent, peer-reviewed journals — e.g., Journal of Personality, Journal of Personality Assessment, Journal of Personality & Social Psychology, Journal of Research in Personality, Personality & Individual Differences — for more than 40 years.

I don't disagree that, as a matter of degree, the Big Five is more widely used in the academic community than the MBTI, and I assume Big Five supporters can now point to more published studies than MBTI supporters. But Stromberg's claims that the MBTI has been all but ignored (and/or affirmatively rejected) among professional psychologists — and "has about as much scientific validity as your astrological sign" — are way off base.

There are hard sciences, soft sciences and pseudosciences and, unlike astrology, temperament psychology in any of its better-established varieties (including both the Big Five and the MBTI) belongs in the "soft science" category, as further discussed in this post, which includes links that point to quite a lot of scientific support for the MBTI.

What's more, the MBTI really doesn't belong in a substantially different category than the Big Five when it comes to reliability (as already discussed) and validity. The 2003 Bess/Harvey/Swartz study I also link to in that last linked post summed up the MBTI's relative standing in the personality type field this way:

Originally Posted by Bess/Harvey/Swartz In addition to research focused on the application of the MBTI to solve applied assessment problems, a number of studies of its psychometric properties have also been performed (e.g., Harvey & Murry, 1994; Harvey, Murry, & Markham, 1994; Harvey, Murry, & Stamoulis, 1995; Johnson & Saunders, 1990; Sipps, Alexander, & Freidt, 1985; Thompson & Borrello, 1986, 1989; Tischler, 1994; Tzeng, Outcalt, Boyer, Ware, & Landis, 1984). Somewhat surprisingly, given the intensity of criticisms offered by its detractors (e.g., Pittenger, 1993), a review and meta-analysis of a large number of reliability and validity studies (Harvey, 1996) concluded that in terms of these traditional psychometric criteria, the MBTI performed quite well, being clearly on a par with results obtained using more well-accepted personality tests.

...and the authors went on to describe the results of their own 11,000-subject study, which they specifically noted were inconsistent with the notion that the MBTI was somehow of "lower psychometric quality" than Big Five (aka FFM) tests. They said:

Originally Posted by Bess/Harvey/Swartz In sum, although the MBTI is very widely used in organizations, with literally millions of administrations being given annually (e.g., Moore, 1987; Suplee, 1991), the criticisms of it that have been offered by its vocal detractors (e.g., Pittenger, 1993) have led some psychologists to view it as being of lower psychometric quality in comparison to more recent tests based on the FFM (e.g., McCrae & Costa, 1987). In contrast, we find the findings reported above — especially when viewed in the context of previous confirmatory factor analytic research on the MBTI, and meta-analytic reviews of MBTI reliability and validity studies (Harvey, 1996) — to provide a very firm empirical foundation that can be used to justify the use of the MBTI as a personality assessment device in applied organizational settings.

And maybe the most important point to stress on the "MBTI vs. Big Five" issue is that, for an ordinary person, there's really no need to choose one or the other. Assuming that the real underlying temperament dimensions that the MBTI is dealing with (in its imperfect way) are the same as four of the dimensions that the Big Five is dealing with (in its imperfect way), I don't see any reason not to look to respectable Big Five sources and respectable MBTI sources (as I do) for interesting data and possible insights into the nature of those dimensions.

In his final paragraph — the same one that tells us that the MBTI "has about as much scientific validity as your astrological sign" — Stromberg also tells us that "thousands of professional psychologists have evaluated the century-old Myers-Briggs, found it to be inaccurate and arbitrary, and devised better systems for evaluating personality." Thousands! Yikes. If he'd just said "hundreds," I'd say there's no way he could come close to backing that assertion with a list of sources. The fact that he found it appropriate to refer to "thousands" of evaluations arguably tells you all you need to know about his fastidiousness in the factual-accuracy department.

It's enough to make you wonder where the man got his "formal training" in journalism.