8. Direction: how far can SD honestly go?

I'm new here · 4/6 I'm working on SD itself · 4/6 I'm working on the backtester · 4/6 I want to rerun the evidence · 3/5

5 min read

Chapter 2 found that SD's squeeze and momentum readings call direction no better than a coin flip once the market's upward drift is removed. A fair reply is: maybe you didn't look hard enough. This chapter looks as hard as we know how, and reports the ceiling.

The short version: SD now calls direction right 56.85% of the time on 2019–2026 data (classic SD: 50.93%), which matches the ~56% estimate. But that 56.85% is the market's drift, not foresight. "Always up" scores exactly the same, and nothing beat it.

#The search, spread across time

A search run once on one train/test split can get lucky: one era happens to suit one idea. So sd_direction spreads the search across nineteen test years:

1. For each test year Y from 2008 to 2026, every candidate model trains only on rows whose outcome was known before 1 January Y (an expanding window, purged at the boundary).
2. It is graded on year Y.
3. A candidate is chosen for year Y by its average record over the five years before Y. That chosen candidate's grade on Y is the honest score. You only ever trade the model you would have picked at the time.

Why choose by the previous five years? Choosing by the full-history table would quietly use the answers. The walk-forward choice is what a careful person could actually have done each January.

#What we threw at it

src/direction.rs builds 41 measurements in 10 families. Each one is there because the finance literature says it carries some direction information:

The candidates are 37 linear models (logistic regression at two penalty strengths) plus three families of small gradient-boosted trees, which can find regimes like "falling market and fear spiking".

#The results

Hit rate pooled over every test year from 2008 to 2026, at four horizons:

Reports: reports/direction_{5d,21d,63d,excess_21d,excess_5d}.txt.

Three things to notice:

• Models that dare to say "down" lose. Their down calls are right 35–45% of the time. Bear markets are rare and different from one another, so a model trained on the last one misreads the next.
• The rank skill is real but tiny. The best per-year AUC (the chance a random up row scores above a random down row) is about 0.52. Higher P(up) does mean slightly more ups, but not enough to flip a call against drift.
• Weekly and monthly SD add nothing to direction (AUC 0.50–0.505). This answers chapter 6's open question about Extended SD.

#What shipped

SD's side of 0.5 now comes from the best call the search could support: P(up) from drift plus the market's weather (SPY trend, momentum, volatility ratio, 5- and 21-bar returns). It is a five-input logistic model, stored as side in src/sd2/sd2_model.json and fitted by sd_direction --ship. How far SD sits from 0.5 still measures the expected move, scaled by time-decay strength:

SD = 0.5 + 0.5 · side · strength · intensity

sd_backtest verify on 2019–2026 data:

Taking momentum's coin-flip lean out of the distance from 0.5 made the expected-move signal cleaner as well.

What this means for you. SD above 0.5 now mostly means "the market drifts up, and here is how big the move will be." Read the distance, not the side. If you want direction you can trade, chapter 11 shows the evidence: no SD direction strategy beat simply owning the stocks.

#Recipe: try your own direction idea

1. Add a column to direction_frame in src/direction.rs, and add its name to a group in GROUPS (or to a new group).
2. Add a candidate in candidates() in src/bin/sd_direction.rs.
3. cargo run --release --bin sd_direction -- --h 21 (and --h 5, --h 63, --excess).
4. Believe it only if the walk-forward choice row beats "drift only". The hindsight table is there to explain, not to pick from.