Over the past year I built an operating system for quantitative trading research: a harness that lets AI agents run the grunt work of strategy development end to end, inside rails that make self-deception structurally difficult. It recently carried two strategies from an empty folder through the full gauntlet: one crypto trend follower to a deployable, drawdown-controlled config, and one options strategy to an honestly sized, validated edge. Along the way it rejected far more ideas than it accepted. This piece is a tour of how it works and why most of the engineering went into saying no.
01 · The problemBacktests want to lie to you.
The hard part of quant research is not finding strategies that look good. It is finding out, cheaply, which good-looking strategies are fiction.
The first “successful” backtest my crypto trend strategy produced returned +42.6%. Champagne numbers. Then a month-by-month decomposition showed that every dollar of profit came from a single January rally; the equity curve went sideways for the five months after. The headline was one great month wearing a five-month flat tail as a disguise.
That experience set the design principle for everything that followed. Modern AI agents can generate strategy ideas, write the code, and run the backtests faster than any human. Generation is now cheap. What remains scarce is refereeing: the discipline that separates a real edge from a lucky sample, an execution bug, or a metric that quietly rewards the wrong thing. So I built the referee.
The harness is a set of protocols, skills, and enforcement scripts wrapped around QuantConnect’s LEAN engine, the same platform available to any retail quant. An AI agent does the building and analyzing. The rails decide what counts as evidence. And a small number of decisions never leave the human: what the strategy is, how much money the research may spend, and whether the result is good enough to trade.
02 · The spineEvery idea walks the same gauntlet.
No experiment influences a decision unless it survives this loop, and the loop is enforced by tooling, not willpower.
Two details in that diagram do most of the work. The review gate means no backtest whose output could influence a decision ever runs on unreviewed code; a launch script literally refuses hashes that have not been blessed. That trades speed for correctness on purpose: wasted compute is recoverable, a wrong verdict is not. And the recorded loser path treats a credible “this does not work” as a research success with its own permanent ID, because the most expensive experiment is the one you unknowingly run twice.
03 · The architectureWho does what: human, agent, rails.
The design question was never “can an AI do quant research?” It was “which decisions must it never make alone?”
This split is also what makes the workflow trustworthy to hand to someone else. State lives on disk, not in anyone’s head: every phase ends with a versioned kickoff prompt for the next one, so a fresh session (or a fresh person) can pick up mid-campaign and know exactly what is locked, what is open, and what has already been tried. Roughly twenty-five phases of one strategy ran across twenty-two separate working sessions without losing the thread.
04 · The tool surfaceWhat the agent can actually reach.
A research agent is only as good as the instruments on its bench. Mine spans a local repo, a cloud backtest engine, a research notebook, and live options market data.
The instruments matter less than the rule that governs them: every one of them writes its output to disk in a form the next session can audit. A notebook that only prints to a screen is useless to a campaign that spans weeks, so results come back as structured log lines and files, reconciled against counters that must agree. When the numbers do not reconcile, the extraction fails loudly rather than handing back a plausible subset.
05 · Three refereesThe checks that changed real verdicts.
Abstract rigor is easy to claim. These three mechanisms each overturned a conclusion I would otherwise have shipped.
- Metrics lie Cross-examine every ratio. A trailing-stop variant once “won” on the pre-registered bar: higher Sortino, shallower drawdown. The equity curve told the truth: it sat below buy-and-hold 99% of days and ended 18% poorer. It was flat in cash a third of the time, and those zero-return days shrank Sortino’s denominator, rewarding timidity rather than skill. The harness now cross-checks Sortino against Sharpe and information ratio and reads the curve before believing any single number.
- Confounds Judge each component on its own evidence. An options strategy’s realized P&L said its entry signal was worthless; the honest read was nearly a project-killing halt. But realized P&L blends entry, exit, and instrument. Re-measured on the underlying’s forward return, before any exit policy touched it, the entry signal was real and broad: a +100% profit cap had been amputating the right tail. The verdict flipped from halt to iterate, and that decomposition discipline is now a standing rule.
- Execution truth Validate the plumbing before the alpha. For weeks the crypto strategy quoted +139% returns. Then a ledger reconciliation revealed the platform had been silently granting 1x buying power where the sizing logic assumed 2.5x, so 63% of intended entries never executed, and the failures were accidentally flattering. The honest baseline was +81.6% at a deeper drawdown, and every prior comparison had to be re-judged. Now an invariant reconciles every intended order against fills or logged rejections, every run.
Notice what these have in common: none is exotic math. They are the unglamorous questions a careful practitioner asks, encoded so they get asked every time. Retail quants do not lose money to missing PhDs; they lose it to headline numbers nobody interrogated.
06 · CompoundingLosers are assets.
Most research pipelines only remember their winners. This one gets faster because it remembers everything.
The same philosophy governs spending. Cloud compute is real money, so the harness maintains an explicit policy for when to pay for parallelism: if a human is waiting on results, batch the runs into a paid parallel wave, because human hours cost more than compute credits; if the campaign is running unattended overnight, queue on the free node, because wall-clock is nearly free at 3 a.m. A 111-name cross-sectional study, one backtest per ticker, fans out for about two dollars. Correctness, usability, and speed are not a slogan here; they are a dial with a written policy for where to set it.
07 · Why it mattersTrust is a workflow property.
The lesson underneath all the machinery.
Every mechanism in this harness exists to answer one user-experience question: when a number appears on the screen, why should anyone believe it? The answer is never “because the model is smart.” It is because the code was reviewed before it ran, the bar was set before the result existed, the costs were real, the sample was big enough to mean something, and the ledger reconciles. Those are workflow properties, and workflows can be designed.
That framing travels well beyond my own trading. Any product that shows people portfolio analytics, performance numbers, or research conclusions faces the same problem in a different costume: intelligence is abundant, and trust is the bottleneck. The way through is the same: decide which checks must be structural, which decisions must stay human, and make the honest path the easy path.
The harness keeps evolving; the current work borrows ideas from recent academic systems that run LLM-driven factor discovery, grafting their generation techniques onto these rails. If you are working on the trust problem in quantitative products and want to compare notes, I am easy to find.