Skip to main content Blog Help (new window)
Blog > Posts > The Small Sample Problem
Back to Home
The Small Sample Problem
One of the biggest problems with applying numerical tools to financial markets is that you need an enormous amount of data to be confident, at least in any kind of statistical sense. Nevertheless, both individual and institutional investors often make decisions with small samples of data, and these decisions produce bad outcomes more often than not:
  • Institutions frequently hire and fire managers based upon three years of performance.
  • Individual investors jump into strategies based upon short-term performance instead of understanding what longer-term history has to say about those strategies.
In fact, the small-sample problem permeates many other areas of life as well. On Tuesday morning, I woke up to sports talk radio and TV talking heads arguing that the San Francisco 49ers should switch quarterbacks based upon a single Monday Night Football performance by the new guy, Colin Kaepernick. In fact, their coach, Jim Harbaugh, has already said he’ll start the "hot hand" at QB for next week’s game against the New Orleans Saints. 
So how can we stop being susceptible to this problem on financial matters? It’s not easy but a good starting point is understanding just how much data you need to be statistically confident. I’ll illustrate this with a real world example and a hypothetical example. If you go over to Ken French's data library site (a popular finance nerd hangout by the way), you can download the annual historical returns for the U.S. equity premium, which start in 1927. If an investor actually had this data starting in 1927 and received updated annual returns every year, it would have taken 28 years of returns data before our investor could have been statistically confident that the reward for owning a diversified portfolio of stocks instead of high-quality bonds was greater than zero. This is virtually an entire investing lifetime.
For our hypothetical example, let’s say we have an investment manager who has outperformed her benchmark by 2 percent per year with volatility of 4 percent per year. This would certainly be respectable benchmark outperformance, but we’d still need to see that occur over 16 years to statistically say the manager was skilled. Even then, we couldn’t be sure of whether we simply found one of the “lucky” fund managers or whether the manager possessed true investment skill.
So the lesson here is to be very, very careful at making any major investment decisions based upon anything less than, say, 10 years of data. I know this is the exact opposite of the way the financial world works, but you’ll likely be better off over the longer term basing your allocation and other investment decisions on longer-term data and sticking to the plan.
Random Links and Commentary of the Week
First off, Happy Thanksgiving to all of you following the blog. I think it’s up to about 10 people excluding family at this point.
I was recently up in Albany, New York at the airport where — amazingly — I counted something like 15 TSA staff in the security line area with exactly ONE PERSON checking boarding passes even though the security line had about 200 people in it. I then discovered, though, that the folks running the airport had come up with a great way for people to blow off steam after spending two hours going through the security line:
This makes a lot more sense than just adding another person or two to check boarding passes, right?
Jared Kizer is the director of investment strategy for The BAM ALLIANCE. See our disclosures page for more information.


There are no comments yet for this post.
Items on this list require content approval. Your submission will not appear in public views until approved by someone with proper rights. More information on content approval.


Body *