9. Inside the backtester

I'm working on the backtester · 3/6

9 min read

Chapter 4 told you what the backtester found. This chapter opens the case and shows you how it works, gear by gear, so you can add an experiment without breaking the one rule that matters: you may only look back.

We follow one imaginary symbol, call it ACME, from a CSV file on disk to a line in reports/indicator_study.txt, and then do the same for an options trade.

 data/history/ACME.csv
        │  history.rs::load
        ▼
 Series { symbol, bars: Vec<DatedBar> }          the truth, never touched again
        │  research.rs::add_noise  (optional)
        ▼
 Vec<OHLCV>  "what SD sees"                       maybe shaken
        │  sd2/features.rs::FeatureFrame::compute
        ▼
 FeatureFrame  (one column per measurement)
        │  research.rs::build_samples   ← outcomes read from the *truth*
        ▼
 Vec<Sample>  (symbol, date, split, x, mag_x, fwd, excess, barrier, …)
        │  sd_backtest.rs  fit_direction / fit_magnitude   (Train only)
        ▼
 Logistic / Ridge
        │  research.rs::scorecard / auc / spearman          (Test only)
        ▼
 one printed line

#Gear 1: the data (history.rs, sd_fetch_history)

sd_fetch_history downloads daily bars from Yahoo and writes one CSV per symbol into data/history/ (git-ignored; 125 files, ~25 years each). Two decisions here shape everything downstream:

#Gear 2: shaking the data (research.rs::add_noise)

If noise > 0, a copy of the bars is perturbed before SD sees it. The real bars are kept, untouched, for grading. This is the most important sentence in the backtester, so here it is again in code:

let clean: Vec<_> = s.bars.iter().map(|b| b.bar.clone()).collect();
let seen = add_noise(&clean, noise, &mut rng);            // SD's eyes
build_samples(k as u16, &s.bars, &seen, cfg, &market, &mut out);
//                       ^^^^^^^ truth  ^^^^^ signal

Each symbol gets its own seeded random stream (seed * 1_000_003 + symbol_index), so runs are reproducible and adding a symbol does not change the noise every other symbol sees.

Why measurement noise and not "random walks"? We are asking "does SD survive bad data?", not "does SD work on fake markets?". Generating fake markets would test whether SD finds patterns we put there. Shaking real markets tests the thing we care about.

#Gear 3: features (sd2/features.rs)

FeatureFrame::compute(bars, cfg) turns bars into columns: volatility, compression depth, the squeeze clock, energy, the eight momentum gauges, the decay curve. Every column at bar i uses bars 0..=i only. That is the first place look-ahead could sneak in, and the reason every rolling tool in primitives.rs is written as a trailing window.

FeatureConfig is the experiment switchboard: --classic-comp and --ao-flip swap in v1's compression or flip rule while keeping everything else, so you can measure one idea at a time.

#Gear 4: samples (research.rs::build_samples)

A sample is one (symbol, day) with everything known at that day's close (x, mag_x) and everything that happened afterwards (fwd, excess, barrier). The outcomes are always read from truth:

Rows are skipped during warm-up (FeatureFrame::WARMUP = 260) and whenever any input is not a finite number. A model fed one NaN learns nonsense silently; it is cheaper to drop the row.

#The purge

The split is by date: before 2014 is Train, 2014–2018 is Valid, 2019 onwards is Test. A sample dated 20 December 2013 has a 63-bar label that ends in March 2014, so it would be trained on using prices from the validation era. build_samples drops any sample whose longest label crosses its era's edge. That costs a few weeks of data per symbol per edge and buys the guarantee that no number in Test was seen during training.

Why three eras and not cross-validation? Shuffled cross-validation mixes 2020 into the training set used to grade 2019. Markets have memory (volatility clusters for months), so that leaks. Time only runs one way; so do our splits.

#Gear 5: models (sd2/model.rs)

Two small models, written from scratch so there is nothing to install and nothing hidden:

• Logistic: "what is the chance this goes up?" Fitted by IRLS (a few rounds of weighted least squares) with an L2 penalty λ that keeps weights small unless the data insists.
• Ridge: "how big will the move be?" A straight-line fit with the same kind of penalty, solved with a Cholesky decomposition.

Both standardize inputs first (Scaler), so a feature measured in dollars doesn't outshout one measured in fractions.

symbol_weights gives every symbol equal total weight: SPY's 25 years must not drown out a stock with 8. Otherwise the model would mostly learn "what old index ETFs do".

Models are fitted on Train (and, for the shipped magnitude model, Train + Valid). Thresholds, like "only grade the top 10% most confident calls", are chosen on Valid. Test is graded once.

#Gear 6: scorecards (research.rs::scorecard, auc, spearman)

scorecard(strength, ret, threshold) grades calls. A positive strength is an up call, negative a down call, and only rows with |strength| ≥ threshold count. It returns the columns you see in every report:

n 412345  cov 100.0%  hit 50.9%  base 56.9%  lift −6.0  bal 50.2%  long 52.1%  ρ +0.003

The column that protects you is base: the hit rate of "always say the majority side" on the same rows. If hit − base (lift) is not positive, the forecast is worse than a broken clock that always says up. Balanced averages the up-call and down-call hit rates, so drift can't inflate it.

Why not just report accuracy? Because stocks drift up. In 2019–2026, "up" was right 56.9% of the time at 21 days. A forecast that says "up" 95% of the time scores ~57% and has learned nothing. Every number in this book is reported next to its base rate for that reason.

#The commands, and what each is for

#The options side (sd_options, strategy/)

The options backtester reuses gears 1–3 and replaces the rest with a day-by-day simulator (strategy/sim.rs::run). For each symbol:

1. Compute SD v2 for every bar from the (maybe noisy) history: sd2_series.
2. Build an implied-volatility series: a real Cboe index for SPY, QQQ, DIA, GLD and USO; the fitted proxy from clean realized volatility for everything else.
3. Walk forward one day at a time. When flat, ask the strategy enter(ctx). When in a trade, reprice every leg with Black-Scholes (pricing.rs) and ask exit(ctx, open). Expiry forces an exit.

What the simulator enforces, so a strategy can't cheat by accident:

sd_options then groups trades by era and IV source (real / proxy) and prints count, mean, median, win rate and worst trade for every strategy next to its Always* twin, the same trade opened on a timer. That twin is the options version of the base rate: if your filter doesn't beat trading blindly, it hasn't earned anything.

#Recipe: add an experiment

Say you want to know whether volume during a squeeze predicts size.

1. Measure. Add a volume_ratio column to FeatureFrame (features.rs): trailing-window only. Append it to FEATURE_NAMES and row().
2. Look. sd_backtest study --diag prints its AUC per target, Train next to Test. For size, add it to MAGNITUDE_NAMES / magnitude_inputs and run sd_backtest magnitude: the ablation lines show how much ρ drops without it.
3. Judge. It must help in both eras. One era is a story; two is a hint; Test confirming Train is evidence.
4. Ship, if it earned it. fit → rebuild → verify → noise → sd_options. Copy changed constants into the ThinkScript study.
5. Write it down. Save the output in reports/, and add a paragraph to the relevant chapter, including if it failed. Negative results stop the next person from trying the same thing.

If you add a new command instead, give it a Some("name") => … arm in sd_backtest.rs::main, reuse dataset() to get samples, and grade with scorecard so its output is comparable with everything else.

#Things that will bite you

• Using fwd in a feature. Anything in Sample after mag_x is the future. Features come from FeatureFrame only.
• Choosing a threshold by looking at Test. Then Test becomes Valid and you no longer have a grade. Pick on Valid (or n<2019 columns in sd_options --sweep).
• Changing train_end() / valid_end() casually. Every saved report assumes them. If you move them, rerun everything and say so.
• Forgetting the seed. A noisy run without a fixed seed can't be reproduced, so it can't be argued with.

Next on this journey: direction, the search for how much direction SD can honestly call.