Insensitivity to Sample Size

When making a decision based on probabilities, make sure to consider the sample size of information the probabilities are drawn on. For example, consider the following problem:

A certain town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. About 50 percent of all babies are boys. However, the exact percentage varies from day to day. Sometimes it may be higher than 50 percent, sometimes lower.

For a period of 1 year, each hospital recorded the days on which more than 60 percent of the babies born were boys. Which hospital do you think recorded more such days?

• The larger hospital
• The smaller hospital
• About the same (that is, within 5 percent of each other)

According to Tversky and Kahneman, most people judge the probability of obtaining more than 60 percent boys to be the same in the small and in the large hospital, presumably because these events are described by the same statistic and are therefore equally representative of the general population.

However, the reality is that the smaller hospital is much more likely to stray from 50 percent. This is because the smaller sample size results in greater variability than the larger sample size. Keep this in mind when presented with information based on a small sample size, whether that is over time or number of occurrences.

This is especially important in distinguishing between luck and skill. For example, consider the hedge fund manager who outperforms the market three straight years. This is less probable to be attributable to skill compared to the hedge fund manager who outperforms the market over twenty years. Again, this is because variability from the mean (in this case the market) is greater over a shorter time frame or fewer number of occurrences compared to a longer time frame or greater number of occurrences.

So how can we test this? One way is if the person can fail intentionally. If a person can fail intentionally, then chances are greater performance is due to skill, not luck.

Overall, be wary when someone presents information or probabilities based on a short time frame or small number of occurrences.

-Joe


source: http://psiexp.ss.uci.edu/research/teaching/Tversky_Kahneman_1974.pdf