4. The steady backtester

I'm working on the backtester · 1/6 I want to change the indicator · 3/4 I want to rerun the evidence · 2/5

6 min read

A backtest is a time machine with one rule: you may only look back. Most backtests that look brilliant broke that rule somewhere without noticing. This one is built to make the rule hard to break, and to be steady: the same command prints the same numbers every time.

cargo run --release --bin sd_fetch_history          # once: 125 symbols, ~25y daily
cargo run --release --bin sd_backtest -- study       # direction, every target
cargo run --release --bin sd_backtest -- grid        # hand-built setups, both eras
cargo run --release --bin sd_backtest -- anatomy     # move size across squeeze life
cargo run --release --bin sd_backtest -- magnitude   # does each idea help size?
cargo run --release --bin sd_backtest -- fit         # freeze the v2 model
cargo run --release --bin sd_backtest -- verify      # shipped v2 vs classic, 2019+
cargo run --release --bin sd_backtest -- noise       # Gaussian noise sweep

Saved outputs from the last run are in reports/.

#The data

sd_fetch_history downloads daily bars for the research universe (research.rs::UNIVERSE): broad index ETFs, every sector ETF, mega caps, financials, health care, consumer, industrials, energy, utilities, REITs, and a handful of high-volatility names. Laggards (INTC, F, KHC, AAL) are included on purpose: a universe of only winners teaches an indicator that "up" is always right.

Bars are dividend- and split-adjusted (history.rs), so a $1 dividend does not look like a $1 crash to the squeeze detector. Dates are kept, so we can split by time.

Survivorship. Every symbol exists today, so companies that went to zero are missing. That flatters "always up" more than it flatters SD, which is one more reason we always compare against "always up".

#Three eras, never mixed

At each edge, any sample whose longest label (63 bars) would reach into the next era is dropped. Without that "purge", the last weeks of training would quietly contain the first weeks of the test.

#Gaussian noise: shaking the data

You asked a good question: does SD only work on the exact historical prints? Real data has bad ticks, late prints and different vendors' closes. So research.rs::add_noise perturbs the history SD sees:

 each bar × exp(ε),   ε ~ Normal(0, (level · σ)²)
 high and low get an extra independent wobble, so the bar's shape changes too

σ is the symbol's typical daily move (a robust median-based estimate), so level = 0.5 means "every print is wrong by about half a normal day". That is far worse than any real data feed. Outcomes are always graded on the real, clean future. Every noisy run uses a fixed seed, and the sweep repeats five seeds per level so the spread is visible.

#How a call is graded

Targets tried for direction: raw return over 5, 21 and 63 days; return minus SPY's (drift removed); and first touch, meaning which of ±1.5 ATR is hit first within 21 days. First touch is the purest form of "which way does the squeeze break?".

#The 75% question

The goal set for this work was 75% directional success. Here is the honest account of trying.

What we tried (all in sd_backtest, all graded on 2019–2026 only after fitting on 2001–2013):

1. Classic SD's own calls: 50.5–51.2%.
2. A logistic model on 16 squeeze + momentum features, plus 3 market-regime features from SPY: AUC 0.48–0.51 on every target.
3. That model's top 1%, 3% and 10% most confident calls: no better, and at long horizons worse. Relationships learned in 2001–2013 partly reverse after 2019.
4. Every hand-built squeeze setup (grid): 49–53% in both eras.
5. Drift-removed and first-touch targets: 50–51%.

Where "75%" can be found, and why it is a trap. In some slices, "always up" over 63 days is right 75% of the time. A model that learns to say "up" in those slices will report a 75% hit rate. That is drift, not skill, and it collapses in the first bear market. The lift and balanced columns exist to catch exactly this.

Conclusion: with daily bars and price-derived features, SD's direction is not predictable beyond a coin flip in this universe, and we would rather say so than ship a number that only looks good. What is predictable is the size of the move. That is where SD v2 put its effort, and what the options strategies in chapter 5 trade.

#Size: where SD earns its keep

verify grades the shipped v2 against classic on 2019–2026, and noise repeats it under noise (reports/indicator_noise.txt; mean ± spread over five seeds):

Even when every single print is off by a full normal day's move, v2 keeps three quarters of its skill at sizing moves. Direction stays at a coin flip (about 50.3–50.4%) at every noise level, for both versions.

#A story about noise: the volatility guard

The first options backtest under noise looked bad: at noise 1.0, the straddle strategy fell from +9.8% per trade to −2.8%. The cause was simple once found. Noise makes one-day returns look bigger, so SD's 20-day volatility rose, so its expected move in percent rose, so it "saw" cheap options everywhere and bought far too many straddles.

The fix uses one fact about measurement noise: it does not add up across days. A bad close adds the same error to a 1-day return and to a 5-day return, but the 5-day return carries five times the real variance. So features.rs computes vol_robust from overlapping 5-day returns. On its own it was worse on clean data, because its 60-bar window remembers the volatility from before a squeeze. The shipped rule is the smaller of the two:

 move_vol = min(σ₂₀, σ_robust)      noise can only inflate σ₂₀, never deflate it

That is 1.3 points given up on clean data for robustness at every noise level. SD_VOL=vol20 or SD_VOL=robust reproduces either column.

#Adding your own experiment

1. Add a feature in features.rs (and to FEATURE_NAMES or research.rs::MAGNITUDE_NAMES).
2. Run sd_backtest study --diag: does it have univariate AUC away from 0.5 in both Train and Test?
3. Run magnitude: does dropping it hurt? (The ablation lines.)
4. Only if it survives: fit, rebuild, verify, noise, then sd_options (chapter 5).

If it only works in one era, it is a coincidence with good manners.