# Why past backtest results mislead

> Overfitting, too many trials, survivorship, hindsight, and changing markets, and how to guard against each.

A backtest shows what would have happened. It's easy to build one that looks
great and then disappoints. These are the usual reasons, and what you can do
about each.

## Overfitting

Every number you tune to the past, such as a 150-day average instead of 200, or
one more condition, makes the past look better. Part of that improvement is the
rule fitting noise that won't repeat.

Warning signs: the result changes a lot when you nudge a number, the strategy
has many conditions, or a rule only makes sense in hindsight.

## Trying too many ideas

Try enough variations and one will look excellent by luck alone. The best of
fifty attempts is not as good as it looks. Keep count, and be more skeptical the
more you have tried.

## Survivorship

The ticker list holds what trades today. Companies that failed and funds that
closed are missing, so a rule that picks individual stocks from today's list is
picking only from survivors. Funds that track broad indexes are much less
affected.

## Hindsight

Tactfolio's readings use only prices up to the day they are read, and trades
happen at that day's close. Hindsight can still sneak in through your own
choices: picking a fund because you know it did well last decade, or adding a
rule because you remember a crash it would have dodged.

## Changing markets

A rule that worked while one kind of asset boomed may not work when that stops.
Gold sits on the defensive side of three of the four starter patterns, and gold
did very well in both decades they were tested on. Part of those results is gold,
not the rule.

## Costs and taxes

Backtests charge a trading cost but not taxes or commissions. Frequent trading
in a taxable account can turn a good backtest into an ordinary result.

## How to guard against it

- **Keep rules simple.** Prefer few conditions and round, sensible numbers.
- **Test another period.** The starter patterns were chosen on 2016–2026 and
  then checked on 2006–2016, a separate decade that includes the 2008 crisis.
  Only rules that improved drawdown and Sharpe in both were kept.
- **Hold out recent years.** Tune on older data, then check once on the most
  recent years. AI assistants can do this for you; see
  [Use Tactfolio from your AI assistant](https://tactfolio.com/docs/mcp.md).
- **Nudge the numbers.** If 150 or 250 days works much worse than 200, be
  suspicious of 200.
- **Raise the costs** and see whether the edge survives.
- **Expect less.** Assume real results will be worse than the backtest.

## Learn the ideas behind it

- [Backtesting](https://tactfolio.com/learn/backtesting-explained.md): What backtesting is, how to backtest a trading strategy step by step, the biases that fool people, and why out-of-sample testing matters.
- [Monte Carlo simulator](https://tactfolio.com/trading-tools/monte-carlo-simulation.md): Run 10,000 random futures for your savings and see the chance your money lasts, with a bad case and a good case.
- [Trading signals](https://tactfolio.com/learn/trading-signals.md): What a trading signal is, how rule-based signals differ from paid tip services, and what the CFTC and SEC warn about signal sellers and pump and dumps.