SIMALTO-Conjoint Comparison

The main historical alternative modelling research method to SIMALTO has been provided by forms of the conjoint method.

While RFT SIMALTO and Conjoint methods both provide similar “outputs” e.g. which combination of benefits, optionally at what price, will be most preferred, they approach this problem from opposite ends of the spectrum.

SIMALTO records many observations on each option:

Is it unacceptable?
Is it a priority need?
Is it expected?
Does a rival have it?
Is it worth paying extra for and if so how much?

etc.

It then builds up a picture of an individual’s relative liking and value between the alternative options so that it can deduce the probabilities of his choosing any combination of them, versus an alternative combination.

Conjoint starts at the end of the SIMALTO process.  It measures relative preferences between preset option combinations (products) and then attempts to deduce the contribution of each individual option to this product preference.

The computer analysis used in conjoint approaches will usually find the very obvious influential options, but its reliance on often-suspect assumptions mean it is increasingly unreliable at measuring the more subtle influences.  In essence there is just too little information gathered, reliably, for unravelling all the values involved – unless there are very few attributes and levels, e.g. up to 5 of each, say.

Consider the following example with 7 attributes and 3 levels of each, named 1,2,3 in order of increasing benefit to the customer:

A typical conjoint presentation to the respondent may be to ask him to give marks out of 10 to each of the following 3 products – described by the levels of benefit on each attribute.

Attribute

Product A

Product B

Product C

1

3

2

2

2

2

3

3

3

3

3

2

4

2

2

2

5

1

2

3

6

1

1

2

7

3

2

1

Marks/10

Preference

6/10

8/10

5/10

 Product B is preferred to Product A.  (In Discrete Choice Models, B may the single chosen product.)  It is only on Attributes 2 and 5 that B is better than A, so it must be something on these two features alone that cause this respondent to rate B above A, (to choose B).

But the computer (analysis) does not know if this preference is 100% due to attribute 2 and 0% due to attribute 5, or 100% due to attribute 5, and 0% due to attribute 2, or some percentage share between both 2 and 5.  Clearly a lot more presentation of alternative products, incorporating different levels of attributes 2 and 5, are needed to resolve this issue, especially if these other products ‘confound’ the comparisons on these attributes with differences on the 5 other attributes.

Reflecting on the difficulty faced by the computer on this problem should indicate to the reader why this approach has natural limitations on the number of attributes it can handle – let alone the respondent difficulty and fatigue in answering the questions needed to resolve such issues.

SIMALTO would have inferred directly if the respondent wanted level 3 instead of level 2 on attribute 2, and 2 instead of 1 on attribute 5, and the degree of priority for any difference in this wanting.

RFT SIMALTO Modelling allows reliable forecasting of both major, and more importantly, the usually more discriminatory secondary issues.  The latter are most likely to be swamped with the data collection and analysis associated with the conjoint method.
Price or cost is dealt with at the individual feature level whereas if conjoint involves price it is at the total product level
RFT SIMALTO provides an independent indication of its simulation forecast accuracy.  Conjoint does not.

Provable links of alternative product/service specifications to:

·        Loyalty

·        Brand Switching

·        Consideration

·        Usage levels (not part of the conventional Conjoint analysis offering)

 

Unlike conjoint, SIMALTO DATA COLLECTION consists of unambiguous real tasks which parallel consumer choice processes

·        What do I have now?

·        What do I expect/desire/want?

·        What is my priority for 'bridging the gap'?

·        What would I pay for these improvement(s)

·        What is its value to me?

·        Will changes impact how I use and/or my loyalty to this product/brand?

This unmodelled data is directly useful in its own right since both respondents and client management understand the key facts whereas in conjoint there is little or no value in the individual data responses, only after their black box improvements.

The pattern-matching approach used by RFT SIMALTO minimises the problems with correlated benefits, since combinations of options are part of the preference evolution questionnaire process

As an expert system, RFT SIMALTO data collection content and analysis rules are bespoke, created for each survey whereas conjoint shoehorns all data from all “suitable” research problems into a fixed equation.  The SIMALTO model is built to best fit each survey’s data and RFT SIMALTO  can demonstrate its accuracy.  Conjoint does not.

Conjoint Limitations

Other researchers often recommend Conjoint as they have no other way of handling complex package trade-off problems.  The better full profile or discrete choice forms of conjoint rely on traditional statistical regression techniques which provide the best mathematical solution to a given set of data.  But they are not cause-and-effect relationships and application to product/price/service preference research can cause problems:

The data collection methodology is difficult.  Respondents tire and give increasingly suspect responses.
There is no direct, easy-to-understand and useful data before it is modelled - the user has to trust the 'black box'.  There are no sense checks.
The utility (value) of an option depends on which other options it is associated with.  In traditional Conjoint, an option has a fixed average utility - for use in any combination of options
The impact of cost or price on the preference data collection for individual options is unknown.  The respondent may assume all are equal, or allocate in his mind unrealistic relative prices/costs.  The analysis calculation does not know this important influence on preference.
 When there are many variables (i.e. option to option changes) then unravelling the link between the single negative price (cost of delivery) utility to each and all of the many positive benefit change utilities are beyond the capabilities of the regression equations (both due to respondent fatigue, and too many variables for too few "reliable" data points).

The rules of regression are not always the best way of understanding the data.  Regression assumes:

·        Normal distributions (not usually true)

·        Un-correlated variables (not usually true)

And the non-statistical user of its equations may assume:

·        Its predictions are causal (this is not necessarily true - there is no in-built logic in creating its weights (utilities))

When these assumptions are not true, the regression assumptions underlying Conjoint break down.

A leading exponent of Conjoint (Rich Johnson and Sawtooth Software in the USA) did not recommend its use for more than 7 attributes, since the issues to be resolved exceed the capability of the model to solve them reliably.  (Also for more than 7 attributes, respondents' required attention and accuracy diminishes)
The questionnaire and regression can be used beyond 7 attributes, but this increasingly invalidates response quality and the analysis accuracy and reliability.  Splitting the tasks between respondents leads to average utilities being created, which may be sufficient for “broad brush” estimation purposes, but not for detailed product and financial planning.
If the reader is still unsure of the suspect nature of Conjoint methods he is referred to the Sawtooth website.  Several ‘papers’ from Sawtooth personnel and “guests” clearly explain conjoint limitations.  Some quotations from text downloaded in February 2000 follow to summarise Sawtooth’s own concerns (Italics are ours and words in brackets are our comments).

 

Conjoint References

Richard Johnson (on the original pair wise conjoint, 1974)

The greatest strength of the procedure seems to be its ability to generate rather refined predictions from quite primitive data…the cost of this strength is the rather heroic assumption of no interaction among attributes.

 

Joseph Curry (on full profile “card sort” conjoint)

With more than 6 to 8 attributes, respondents can become overloaded with information, and the reliability of their answers may diminish.  When using a card sort, limit the number of attributes to 8 or less

Joseph Curry (on ACA – adaptive conjoint)

It is possible to ask respondents to evaluate more attributes than they would consider when making a purchase.  This can cause the relative importance of attributes to be reduced.  In pricing studies, it can lead to an artificial reduction in price sensitivity.  For pricing studies consider using card sort … (limit 6-8 attributes – see above)

Most conjoint studies account for main effects only … one important exception is brand and price (so brand and price should not be among the attributes?)

Attribute levels must be mutually exclusive and exhaustive

Researchers have found that as the number of levels increases the measured importance of an attribute increases as well

Responding to conjoint questions over the telephone is too difficult for respondents

Stephen Struhl (on full profile conjoint, 1994)

While conjoint analysis solved many of the problems that arose with earlier methods, it also began showing some shortcomings.  (Hence Discrete choice) Brand name itself … cannot be traded accurately against other attributes.

Problems can arise in asking respondents to rate rank product profiles.  Sometimes rankings prove difficult.  Looking at a series of 16 product profiles, most of us probably could select the one we liked best and the one we liked the least.  It is harder to decide which should be ranked 5th or 6th.  Unfortunately, accurate rankings or ratings of all the product profiles are needed for conjoint analysis to provide meaningful results.  Discrete choice modelling is still much easier than the complex gyrations required when splitting conjoint designs which tend to become messy and produce less than ideal results.

 

Stephen Struhl (on discrete choice modelling)

Discrete choice modelling has one salient limitation.  Analysis can be done at the aggregate level only (i.e. average respondents)
Discrete choice modelling remains far more difficult than conjoint when it comes to analysis and model specification
While clustering models usually converge, or behave, discrete choice models may not
Several factors can cause non-convergence with discrete choice modelling:
Multicollinear variables (variables highly correlated with each other)
The presence, among the choices in the scenarios of one of more alternatives selected very infrequently
The presence in the design of any highly infrequent variable or variables.
Unfortunately it is unclear how infrequent a variable or an alternative can be without causing problems
If five products appear in all the discrete choice model scenarios, you cannot do accurate estimation of what might happ en if there were only four, or if another product entered the market as a sixth competitor.
Linear regression (used in full profile) definitely does not work correctly when you are trying to predict a variable that can take only two values, such as ‘choose versus don’t choose’ … and should not be used when the dependent variable can take only a few discrete values (hence multinomial logic is used for discrete choice modelling).

(The above provides a strong argument against pair wise and full profile conjoint, both for respondents and analysis.  The strange thing to emerge is that while card sort/ full profile is agreed to be often too difficult, etc. the discrete choice modelling method of simply picking the winner (or one of) several alternatives is claimed to be better.  Yes, it may be more like ‘real life choice’, but the analyst has far less information from which to deduce option contribution to choice.  It is true he does not have erroneous information, and no information in discrete choice model may be better than erroneous information from full profile, but basing good predictions on much less information seems in itself hard to believe.  Yet this article claims statistically “in fact, more attributes and levels actually can make it easier to fit a model closely to the data”.  This may be true in a pure statistical truth sense, but seems far from real life given the concerns expressed elsewhere.)

 

HB –Reg for Hierarchical Bayes Regression Sawtooth Software, Inc, Sept 1999

This article introduced a method on enabling conjoint data derived from averaging respondents to be modified so that cluster analysis seeking segments of like minded respondents can be effected.  Pages 4 and 5 summarise problems with standard full profile, discrete choice model methods.

Under certain conditions, regression analysis can provide estimates of the b’s (regression weights = part-worths or importance weights) for the respondent:

1.     Usually we assume the errors are random … and the sum of squared errors is as small as possible

2.     The respondent must have rated at least as many companies as the number of variables for which we seek importance weights

3.     The respondent ratings on different variables must have some degree of independence from one another. 

 Unfortunately, researchers doing customer satisfaction studies usually find none of these conditions is satisfied.

 The first condition is seldom satisfied because respondents tend to bunch their ratings at the top ends of the scale, so random variability tends to become small for highly rated products.

 The second condition is an even more serious impediment to the estimation of individual importance weights.  Respondents get bored when asked to rate many companies on many attributes, and the quality of their output suffers (and respondents may not be knowledgeable about a large number of companies)

 Failure to satisfy the third condition is also a serious problem.  We would like a respondent’s ratings on several variables to be completely independent of one another, but that is almost never true.

(This article then goes on to illustrate how even more complex mathematics can help reduce these problems, and illustrates it with an example having 5 attributes, each at 4 levels – not in our experience a typical customer satisfaction or product design/ price study.)

 


SIMALTO Limitations

While we feel SIMALTO’s completely different approach avoids most of conjoint’s problems, and at least has the massive benefit of creating/using sensible option information in its own right, it too has its limitations.

1.     SIMALTO spends most of its interviewing effort and time researching respondents’ priorities around the specification they have now (or a given start position), and particularly about their personal improvements to this position.  Therefore it is not so accurate in deciding between which options they do not want will make them marginally more/less satisfied. 

     The assumption they are less influential than prioritised options is correct, but it is probably not true that they are all equally less influential. Yet SIMALTO analysis has to assume some probability of less influence – usually pro rata to the cost/ price of unwanted options e.g. including in a product an unwanted $100 option is going to be more detrimental to preference than including an unwanted $10 option, unless they are both ‘free’ extras.  So while SIMALTO logic does its best, influence of options too poor or too good is less well estimated than those in the real choice set.

Our view is that this is a good fault, maximising data quality where it is needed and this has a less serious impact than in conjoint where estimates of such “unwanted” options are also made (from equally “uncaring” respondents) and worse, their presence influences the estimates of options more important to preference choice –see above.

2.     Respondents tend to be much more adept and interested in making improvements  than degradations – particularly in service studies where they do not see a price saving for giving up a benefit they now enjoy, and they realise that the supplier is not really likely to degrade his product anyway.  If possible, respondents should start their priority improvement stages from a ‘lowest common denominator’ position so improvement to possible real life positions are also accurately measured.

3.     If respondents perceive they have a high level of some attributes this may be due to the true situation, or their own optimism about a product they like.  If they only make improvements to their current position we may not accurately know how valuable (important) it was to get from a lower level to their perceived level.  Many clients feel this is justified since the client  invests to improve perception  (the customer  is always right).  But SIMALTO will not have an explicit measure on this unasked-for change from lower (true?) level to higher (perceived?) level, unless the lower was seen as unacceptable, or a saving, or a redesign “trade down”.  So while there are these three chances for such a measure, if asked, a safer option might be to start the respondent from the true level if known, rather than perceived levels, so he/she can “justify” the improvement to the perceived level against rival trade off improvements he could make on other attribute benefits.

Points (2) and (3) above indicate that the study and questions used should be geared to the decisions the client has to make – true for any market research except the “nice to know” variety, which can give market research investment a bad name from disappointed clients.

4.     Particularly in pricing studies, the options included on the grid should be realistic trade offs, e.g. it is not particularly sensible when specifying a large truck, say, to include options such as sleeping compartment, or an extra axle, each costing several thousand dollars, together which options such as cigarette lighter, or better quality carpets in the cab, costing tens of dollars.  There are several ways of dealing with this including:

a.     Making the ‘cheap items’ optional extras – i.e. providing a check list asking which of these would you buy at the prices marked after the main items have been traded off, or

b.     Having two SIMALTO grids – one for the $250+ items and one for the under $250 options, say

5.     The constrained prices/ points budgets used to allocate priorities must be large enough to reach options priced on the grid (or else they cannot be chosen, unless there is a catch all “anything else you want” question).  If some items on the grid are relatively very expensive then a budget large enough to reach these may ‘buy’ the equivalent of many low priced items.  If a respondent does not choose an expensive item, but many cheap ones, then the analyst will not know the relative priorities between the cheap ones chosen within this budget stage – the program assumes they are all equally “important”, unless there are other pieces of information about them.  This loss of sensitivity may be acceptable, but on balance relatively very expensive items should be avoided – treated offline in a scenario setting context, or questioned standalone, or see (4) above.

Budgets must be sufficient to cover the decision objectives of the research.  This may seem obvious, but a case history follows to illustrate this important point.  An airline was considering enhancing its economy class product service.  Two sets of 50-point bonuses did a very satisfactory job of finding and predicting which alternative 50, 60, 70 extra point equivalent packages would meet with most customer approval.  (Note if the investment decision before the study commencement was thought to be a spend of 50-70 points extra, then the bonuses should “bracket” this range.) 

Flushed with success, the client then asked if the data could also be used to decide which of alternative “business class” packages these economy passengers would prefer.  On the grid typical business class packages cost around +180 points – way beyond the +100 (50 + 50) spend economy passengers had made.  (Equivalent to asking compact car drivers to choose between option packs on Jaguar/Mercedes/Lincoln cars).  Disappointment followed with the caveats attached to “guesses” on preference forecasts between alternative +180 point packages.  These guesses could have been reduced if appropriate questions had been included  (e.g. now spend a third bonus of +100 points).

6.     Because options chosen within a budget spend scenario may not be further discriminated, 3 or 4 medium sized budgets will provide better sensitivity than 1 or 2 larger budgets.  Number of attributes/levels and type of interview medium, and the project objectives impact the optimum  balance between number of and amount of each budget scenario.

The reducio and absurdum approach of making one option improvement at a time could get over this size of budget issue, but raises issues of option inter collinearity and repetitious questioning leading to poor quality response.  SIMALTO is meant to be a flexible interviewing approach designable for the problem at hand, and as such experience dictates the balance the questionnaire designer should make to maximise data quality and minimise assumption reliance.

7.     Some clients find it hard to provide the initial seeding grid price/cost points, particularly for service projects.  They fear that errors in estimation will result in erroneous priorities.  This will be true if individual seeding points are relatively very inaccurate (e.g. over double or less than half what they should be) but as long as the error in a cell cost is less than, say, 25% of a bonus spend, and less than say 25% relative to other cell cost estimates, then this should not affect overall priorities too much.

        SIMALTO is not meant to provide “accountancy accuracy” but instead provide a reliable forecast of relative priorities/preferences.  Avoidance of this issue by providing no relative option costs (as in conjoint) is, in our estimation, a potentially much bigger problem, since then we don’t know how respondents are guessing relative option prices in their total product (or even worse, part product) preferences, and we ignore a potentially crucial factor in product/service optimisation.

In Summary

SIMALTO is flexible in its data collection and analysis procedures.  It is not a panacea for all multivariate market research projects.  Indeed simple total concept/price preference is sometimes the best approach when product/option flexibility is limited.  On balance however, when there are 6+ attributes and/or real price/cost differences between options, then its limitations above are acceptable, and minimised with a commonsense approach to the problem at hand.