How “good” is a good guess, statistically
How to guess better as a business.
“Guess” & “estimate” would be used interchangeably throughout this article and mean the same thing in this context.
Guess how much it’ll cost you to build a new house.
If you guessed wrongly, how much extra money would you likely spend?
Find out with good ‘ol statistics.
These questions are forms of cost projections and provide you, the one building the house with a clearer understanding of the cost uncertainty.
Ideally, you’d want to spend less than or exactly what you planned when building your house, right?
How accurate can your guesses be?
Say you want to build a house and need to know how much money you might need for the entire project.
A P10 or Optimistic Estimate is the lowest amount you could possibly spend on building the house. It’s like if everything goes well and you predict the future like Basil Hawkins from One Piece.
A P90 or Pessimistic Estimate is the highest amount you could spend when things don’t go well and your guess was way off. In this case, you’d need more money to complete the house. Not exactly the desired outcome.
The difference between the two estimates is the “accuracy range”, which tells you how much the cost of your new house may vary.
For example, if the P10 estimate of your new house project is ₦10m, and the P90 estimate is ₦25m, the accuracy range is ₦15m. This means you should have at least ₦10m (P10) and maybe up to ₦25m (P90) before you start because the cost could be anywhere in between.
The wider the accuracy range (the bigger the gap between the P10 and P90), the more uncertain you are about the cost. If the range is narrow (P10 and P90 are close), you have a better idea of how much you might need. It’s like knowing if the cost of building a house is usually cheap or expensive so you can plan accordingly.
In business planning involving projects that cost millions of dollars (or billions of naira, sadly), it is important to know if you’d likely spend more or less on these projects before you start them.
Accuracy range estimation in the wild
In business, the accuracy range can be estimated using statistical analysis of offset data — that is, similar or related past projects.
By studying the historical data and applying statistical techniques — often simple ones you learnt in secondary school, one can gain insights into the typical range of outcomes seen in similar situations.
This then helps in estimating the potential range of costs for that house you want to build in the village.
Getting started
Let’s look at a sample offset data. This is basically past guesses you or someone else made. Think of this as house prices in million naira.
A perfect guess gives a ratio of 1 (third column).
Statistical analysis of this offset data gives us an accuracy range denoted as a probabilistic range (remember P values).
Now, we visualize this analysis to create some perspective.
Chart A depicts how your guesses hover around the perfect guess (1) using a generic distribution of outcomes. On the x-axis is the number of guesses and on the y-axis is the ratio. The closer a guess (data point) to the line, the more accurate it was.
Chart B depicts the accuracy range using the same generic distribution of outcomes. On the x-axis is the potential value of the outcome of interest, in this case, the cost of a house. On the y-axis is the expected frequency of the potential outcomes (if we make a lot of guesses).
Remember normal distribution?
Most guesses are expected to fall close to the estimate, and this is why the distribution peaks around the estimate. Some guesses could be farther away from the estimate but would tend to occur less frequently.
So how does this help my business?
The accuracy range provides the business planning team with an estimate of the potential range of outcomes and these values are inputs into budgeting and other economic analysis.
The ratio tells us a lot about our estimating process and whether we are accurate, and how much our actuals vary around our estimates.
If our guesses are unbiased, then the Average (mean) of the ratio should be close to 1, as is the case in this example.
For similar houses we wish to build, the P10/P90 accuracy range would be reported as -14%/+24% relative to the estimate.
This means a good guess would be 14% cheaper than the actual cost of the house and a bad guess would be 24% more expensive!
If this analysis is done on your business and your range appears wide (your average mean is far from 1), it indicates a systematic bias in your estimating process (basically, you suck at guessing). It could also mean your sample size is small because the estimate of the Mean is subject to sample variation, or you made one or two bad guesses historically.
In conclusion, the purpose of this analysis is to study the estimating process of the business and its average performance, not the outcome of any one activity.
Thanks for reading 💜
