I let AI build a trading bot, then Reddit caught my overfitting mistake
Overfitting backtest: Two people on Reddit pushed back on my last post about testing 324 parameter combinations. Both were right, and their pushback turned into the most useful test I’ve run on this bot so far. Here’s what they said, what I did about it, and what the corrected numbers actually show including the uncomfortable parts.
- ▪Overfitting backtest: Two people on Reddit pushed back on my last post about testing 324 parameter combinations.
- ▪Both were right, and their pushback turned into the most useful test I’ve run on this bot so far.
- ▪Here’s what they said, what I did about it, and what the corrected numbers actually show including the uncomfortable parts.
Opening excerpt (first ~120 words) tap to expand
Overfitting backtest: Two people on Reddit pushed back on my last post about testing 324 parameter combinations. Both were right, and their pushback turned into the most useful test I’ve run on this bot so far. Here’s what they said, what I did about it, and what the corrected numbers actually show including the uncomfortable parts. The critique After I posted that the best of 324 backtested combinations returned +68.6%, someone pointed out the obvious flaw I’d skipped: testing 324 combinations against the same data and reporting the best one is a textbook multiple-comparisons problem. With enough combinations, something will look great by chance alone, whether or not there’s real signal underneath.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at AI Project Log.
Discussion
0 commentsMore from AI Project Log
