Lonely? Dr. Max Diff to the Rescue! Benefits of the MaxDiff Approach Over Standard Rating Exercises

By Shiv Raman, M.A., M.B.A.,
Senior Vice President and Chief Marketing Scientist, GfK V2

I have a terrific idea for an online business in the brave new world of Web 2.0. Let’s find out what busy, career-driven singles really want in a partner, what’s really important to them and create compatibility metrics to match couples. People would pay anything for such a service, right? It’s foolproof!

OK. Reality check. According to Wikipedia, there were at least 844 lifestyle and dating services at the end of 2004. That number is likely closer to a thousand now. So, not a great business idea in the summer of 2007, but I bring this up for another reason. A quick run through some of the major dating sites reveals that they all require prospective singles to fill out an online questionnaire. The questionnaire is presumably used to develop profiles of the respondents and thereby identify the kind of people they are likely to be compatible with. Many of these questions measure the importance of traits such as intelligence, looks, sense of humor and so on. But, just how useful are such measures of importance? Would anyone rate any of these attributes as being less important relative to the others?

My contrived example illustrates two well-known issues in the measurement of stated attribute importance. First is the obvious point that if our attributes are all of the “mom and apple pie” variety, we are unlikely to detect variation in responses. This is the reason online dating sites want to know whether your idea of a good time involves dodging kinkajous and agoutis in the rainforest or sipping mojitos at a candle-lit table. Second, even if attributes cover a wide spectrum of issues, are such ratings-based exercises capable of identifying the top few attributes that truly matter? This latter point is the premise of this column – that typical, scale-based questions to measure stated importance are seldom a source of great insight. There are many reasons for the poor information content of such tasks, not the least of which are a) variations in how people interpret and use scales (think Grumpy Gordon with his fours and Sunny Sally with her tens) and b) a general unwillingness on our part to admit that some things really are unimportant (Bigger! Better! Faster! More!)

So what’s a poor researcher to do? She can’t just stop measuring attribute importance – if for nothing else, out of compassion for the legions of research agencies that churn out glossy quadrant maps by the dozen, plotting stated and derived importance with circles and arrows and a paragraph under each one. No, the solution must lie in thwarting the twin evils that plague ratings-based importance measurement, namely, scale usage bias and lack of differentiation.

Such a solution does exist in the form of a technique known MaxDiff. In the MaxDiff exercise, respondents do not rate attributes on a scale; instead, they evaluate attributes in smaller groups (typically four to six at a time) and pick the most and the least important in each group. This is an “information-rich” task, since for any subset of four attributes, identifying best and worst allows us to infer the ordering for five out of the six possible comparisons; that is, if attribute A is best and D is worst, we automatically know that A>B, A>C, A>D, B>D and C>D. An experimental design, known as the Balanced Incomplete Block Design, is utilized to ensure that all attributes appear in every position the same number of times to avoid bias arising from context and order effects.

Sample Input