Let the AI Do the Experimenting

Agentic AI Let the AI Do the Experimenting Using autoresearch to optimize marketing campaigns under budget constraints Mariya Mansurova Apr 28, 2026 14 min read Share Image generated by author with DALLE-3 Have you ever been in a situation where you have plenty of ideas on how to improve your product, but no time to test them all? I bet you have. What if I told you that you no longer have to do it all on your own, you can delegate it to AI. It can run dozens (or even hundreds) of experiments for you, discard ideas that don’t work, and iterate on the ones that actually move the needle. Sounds amazing. And that’s exactly the idea behind autoresearch, where an LLM operates in a loop, continuously experimenting, measuring impact, and iterating from there. The approach sounded compelling, and many of my colleagues have already seen benefits from it. So I decided to try it out myself. For this, I picked a practical analytical task: marketing budget optimisation with a bunch of constraints. Let’s see whether an autonomous loop can reach the same results as we did. Background Let’s start with some background to set the context. Autoresearch was developed by Andrej Karpathy. As he wrote in his repository: One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of “group meeting”. That era is long gone. Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures in the skies. The agents claim that we are now in the 10,205th generation of the code base, in any case no one could tell if that’s right or wrong as the “code” is now a self-modifying binary that has grown beyond human comprehension. This repo is the story of how it all began. -@karpathy, March 2026. The idea behind autoresearch is to let an LLM operate on its own in an environment where it can continuously run experiments. It changes the code, trains the model, evaluates whether performance improves, and then either keeps or discards each change before repeating the loop. Eventually, you come back and (hopefully) find a better model than you started with. Using this approach, Andrej was able to significantly improve nanochat. Image by Andrej Karpathy | source The original implementation was focused on optimising an ML model. However, simialr approach can be applied to any task with a clear objective (from reducing website load time to minimising errors when scraping with Playwright). Shopify later open-sourced an extension of the original autoresearch, pi-autoresearch. It builds on pi, a minimal open-source terminal coding harness. It follows a similar loop to the original autoresearch, with a few key steps: Define the metric you want to improve, along with any constraints. Measure the baseline. Hypothesis testing: in each iteration, the agent proposes an idea, writes it down, and tests it. There are three possible outcomes: it doesn’t work (discard), it worsens the metric (discard), or it improves the target (keep it and iterate from there). Repeat: the loop continues until you stop it, improvements plateau, or it reaches a predefined iteration limit. So the core idea is to define a clear objective and let the agent try bold ideas and learn from them. This approach can uncover potential improvements to your KPIs by testing ideas your team simply never had the time to explore. It definitely sounds…

This excerpt is published under fair use for community discussion. Read the full article at Towards Data Science.