A/B testing in everyday life

By dkl9, written 2024-327, revised 2024-327 (0 revisions)


If Pat asks Sam "does this hat look good?", they assume that the hat looks either good or bad. If the hat looks bad, Sam can reply that it does, which may make Pat feel bad — insofar as Pat would instinctively identify with the hat by having considered it — and/or develop some resentment toward Sam's negative comment. So Sam is incentivised to say the hat looks good, however it actually looks, destroying their answer's value as information.

If Pat has two hats to consider, they can ask which one looks better. Short of that, they can ask whether the hat or their bare head looks better. Either way, that kind of comparative question works much better.

When Pat asks Sam to compare, either way Sam could answer would feel acceptable to Pat. Any answer declares the same number of options as good as any other answer. The incentive on Sam, if any, is to inform correctly. With one option, taken as good or bad, "it's ambiguous" looks to Pat like a cop-out. With multiple options, Sam only says "it's ambiguous" when it really is at least a bit ambiguous to compare similar-quality options.

As questions, "is it good?" is absolute and "which is better?" is relative, but as answers, "that one's better" is the less relative way. For the hat's appearance to resolve as good or bad, Sam must pick an imagined, partly arbitrary neutral point. Comparisons avoid that noisy flexibility.

Comparisons fit more easily into Abs-E. Sam would naturally answer a comparative question by naming an option, while "does this hat look good?" is a "yes"-or-"no" question, answered only awkwardly with "good" or "bad".

The hat problem is a special case. Other judgments are more or less emotionally laden, and so the incentives to answer honestly break more or less severely depending on the question. However severely they break, comparing to a specific alternative makes the question fairer. On any topic, picking out specific options to compare gives answers more reliable and positively-worded than asking about the quality of one option.

You may ask yourself "is this essay good enough to share?", but you'd be better off asking instead "which is better, this essay, or the last one I shared?".