mr.piron.Back to the research desk
AI & MACHINE LEARNING

In an AI Factor Search, the Number of Candidates Tried Is Part of the Result

On 100 simulated noise factors with no leakage, a correctly scaled test still leaves six apparent discoveries. A threshold set by the trial count removes all six. In this ten-asset panel a cost filter keeps the best one.

TECHNICAL RESEARCHResearch9 primary sources2,948 words
MR. PIRON / RESEARCH DESK14 min read
010203040506070809123456789ARTICLE, IN READING ORDER →↑ NUMBERED SOURCES010203040506070809123456789ARTICLE, IN READING ORDER →↑ NUMBERED SOURCES010203040506070809123456789ARTICLE, IN READING ORDER →↑ SOURCES
FIG. 0 EVIDENCE MAP · 9 SECTIONS · 9/9 SOURCES CITED · 14 CITATIONS · 8 EQUATIONS · 1 CODE LISTINGComputed from this article. Each bar is a section, as long as its share of the words. A curve joins a passage to a source it cites; a circle grows with the number of passages that cite it. Diamonds mark equations and squares mark code.
Sources cited, equations and code listings per section
SectionSources citedEquationsCode listings
1. What a search over one hundred noise factors shows100
2. Availability clocks and the labels they protect220
3. Purged and embargoed folds for overlapping labels4, 520
4. The trial count as part of the test3, 610
5. Scoring the search itself on the purged folds4, 600
6. From rank IC to weights, turnover and cost730
7. A reproducible program for the worked examplenone01
8. What recent audits and benchmarks report1, 7, 8, 900
9. Assumptions and limitations of this demonstration100

01 /What a search over one hundred noise factors shows

A language model can propose formulaic factors in bulk, and each proposal is one more draw from the distribution of lucky backtests. This article shows, on a simulated market in which no candidate carries any information, that the number of candidates tried belongs in the result itself. No leak is present, the labels do not overlap and the statistic is scaled by its exact null variance. Still, 6 of 100 noise factors pass a one-sided test at the 5 percent level. A threshold computed from the declared budget of 100 trials passes none. Scored out of fold, the search that shows a rank IC of 0.082 in sample earns a figure whose expectation is zero, 0.006 in this draw.

Published systems already report the ingredients. Hubble, an agentic factor-mining framework, describes a main run that evaluates 104 valid candidates across three rounds, after which the top five factors are fixed and validated on a held-out period [1]. That description contains a count of attempts and a selection step, and the count is the quantity this article follows.

Four controls are kept apart because each answers a different error. The availability clock decides what a feature may contain at a decision date. Purging and an embargo decide which training dates may sit next to a test fold. The trial budget sets the significance bar. Turnover and cost decide whether a rank correlation says anything about money. One Python program of 59 lines computes every simulation result quoted below.

02 /Availability clocks and the labels they protect

Index assets by i=1,…,Ai = 1,\dots,A and decision dates by t=0,…,T−1t = 0,\dots,T-1, and write rt,ir_{t,i} for the return of asset ii realized over date tt. A raw field carries two timestamps. Its event date dd says which period the value describes. Its availability time a(d)a(d) says when the value could first have been read. A feature used at date tt may contain only values with a(d)≤ta(d) \le t, so it is built from the latest admissible event date. The label is the return over the next hh dates and never touches a return dated tt or earlier.

Feature built on the availability clockEQ. 2.1
xt,i=vd∗(t), i,d∗(t)=max⁡{ d:a(d)≤t }x_{t,i} = v_{d^{*}(t),\,i}, \qquad d^{*}(t) = \max\{\, d : a(d) \le t \,\}

vd,iv_{d,i} is the raw value of the field for event date dd and asset ii, and a(d)a(d) is its availability time. In the program a(d)=d+La(d) = d + L with a publication lag of L=2L = 2 dates.

Forward label and rank information coefficientEQ. 2.2
yt,i=∑δ=1hrt+δ, i,ICt=corr⁡(rank⁡(xt,⋅), rank⁡(yt,⋅))y_{t,i} = \sum_{\delta=1}^{h} r_{t+\delta,\,i}, \qquad \mathrm{IC}_t = \operatorname{corr}\big(\operatorname{rank}(x_{t,\cdot}),\ \operatorname{rank}(y_{t,\cdot})\big)

The label sums the next hh one-date returns. ICt\mathrm{IC}_t is the Spearman rank correlation, taken across the AA assets, between feature and label at date tt.

The program contains one field that makes the difference measurable. Its value for event date dd is the sum of the LL returns that follow dd, as for a value stamped with its period but finalized only at publication, so a(d)=d+La(d) = d + L. On the availability clock the feature at tt is vt−Lv_{t-L}, a sum of returns already realized, which says nothing about independent future returns. Read by event date, the feature at tt is vtv_t, a piece of the label. On the statistic zz of the next section the first version scores −1.38-1.38 and the second 9.809.80.

The second score exceeds every threshold used in this article. It is not a multiple-testing problem, and no threshold repairs it. The Availability Contract, a recent proposal for earnings-call benchmarks, observes that chronological train and test splits do not prevent a timing mismatch that sits inside a single event. It requires a benchmark to state the input boundary, the evidence of input availability and the start of the outcome window [2].

03 /Purged and embargoed folds for overlapping labels

With h>1h > 1 the labels of neighboring dates share returns: the label windows of dates ss and tt intersect exactly when ∣s−t∣≤h−1|s - t| \le h - 1. Split the TT dates into KK contiguous test blocks, and let block kk run from date ℓk\ell_k to date uku_k. A training date within h−1h - 1 dates of the block, on either side, has a label that shares a return with a test label. Purging deletes those dates. The embargo deletes ee further dates after the block, as a margin against serial dependence that outlasts the label window.

Training dates after purge and embargoEQ. 3.1
Tk={ s:s<ℓk−(h−1) }  ∪  { s:s>uk+(h−1)+e }\mathcal{T}_k = \{\, s : s < \ell_k - (h-1) \,\} \;\cup\; \{\, s : s > u_k + (h-1) + e \,\}

ℓk\ell_k and uku_k are the first and last dates of test block kk, hh is the label horizon and ee the embargo. Every ss in the set satisfies ∣s−t∣≥h|s - t| \ge h for every test date tt, so no label window is shared.

The program counts the pairs of one training date and one test date whose label windows share a return. With T=240T = 240, K=4K = 4 and h=5h = 5 the unpurged split has 60 such pairs, which equals the closed form (K−1) h (h−1)(K-1)\,h\,(h-1). Each interior boundary is crossed from both sides, and gg pairs cross it at distance gg for g=1,…,h−1g = 1,\dots,h-1. After the purge the count is 0, with or without the embargo of e=5e = 5 dates, and it is positive again when the purge is one date short.

A recent theory of time-series validation gives the same accounting. It prices purged kk-fold with an embargo at a cleared zone of one label horizon before each test block and one horizon plus the embargo after it [4]. In the date arithmetic used here that is h−1h - 1 dates on each side and ee more after. A forward-chaining splitter needs only the first half. In scikit-learn, TimeSeriesSplit trains on the folds that precede the test fold and offers a gap parameter that drops samples from the end of each training set [5].

Overlap also distorts the test statistic of a single candidate when its scores persist from date to date. Take as null hypothesis that a candidate's ranks are a random permutation independent of the labels. Each rank IC then has mean zero and variance 1/(A−1)1/(A-1), the classical null moments of Spearman's coefficient without ties. Dates spaced hh apart have disjoint label windows. When returns are independent across dates, as in the simulation, their ICs are independent given the scores, and the standardized mean zz over those n=T/hn = T/h dates has unit variance.

Standardized mean rank IC on spaced datesEQ. 3.2
z=n (A−1)  ⋅  1n∑m=0n−1ICmh,n=Thz = \sqrt{n\,(A-1)}\;\cdot\;\frac{1}{n}\sum_{m=0}^{n-1} \mathrm{IC}_{mh}, \qquad n = \frac{T}{h}

The nn summands use disjoint label windows. Under the null, with returns independent across dates, they are independent given the scores and each has variance 1/(A−1)1/(A-1). Then zz has unit variance and is approximately standard normal.

Across the 100 noise candidates the program measures a standard deviation of 0.92 for zz on the 48 spaced dates. The same scaling applied to all 240 overlapping dates, as if they were independent, gives a standard deviation of 1.60. There, 21 of the 100 candidates clear the one-sided 5 percent threshold where 5 are expected. The inflation needs both overlapping labels and persistent scores, here with an autocorrelation of 0.9 per date. Ranks redrawn independently at each date would give uncorrelated ICs whatever the labels share, a case that the null settles and the program does not run. With persistent scores, a pipeline that counts overlapping dates as independent manufactures significance before any search begins.

04 /The trial count as part of the test

Suppose NN candidates are scored and each zjz_j is approximately standard normal when candidate jj is noise. A one-sided test at level α\alpha passes a candidate when zjz_j exceeds Φ−1(1−α)\Phi^{-1}(1-\alpha), and the expected number of noise candidates that pass is αN\alpha N whatever their dependence. The probability that at least one of them exceeds a threshold cc is limited by the union bound, and solving that bound for level α\alpha gives the Bonferroni threshold cNc_N.

Bonferroni threshold for a budget of trialsEQ. 4.1
Pr⁡(max⁡1≤j≤Nzj>c)  ≤  N (1−Φ(c)),cN=Φ−1 ⁣(1−αN)\Pr\Big(\max_{1 \le j \le N} z_j > c\Big) \;\le\; N\,\big(1 - \Phi(c)\big), \qquad c_N = \Phi^{-1}\!\Big(1 - \frac{\alpha}{N}\Big)

Φ\Phi is the standard normal distribution function, NN the number of trials and α\alpha the family-wise error level. For standard normal zjz_j the inequality is the union bound over NN null candidates, which needs no independence. At c=cNc = c_N its right side equals α\alpha.

In the program N=100N = 100 and α=0.05\alpha = 0.05. The single-test threshold is 1.645 and the expected number of passes is αN=5\alpha N = 5. The draw produces 6, and the best candidate has z=2.45z = 2.45, a discovery for any pipeline that forgets how many candidates it scored. The Bonferroni threshold is c100=3.29c_{100} = 3.29, and no candidate passes it. That outcome belongs to this seeded draw: the bound itself allows a noise candidate above cNc_N in up to 5 percent of draws.

The bound holds for the NN that was fixed before the first result, so the count is a budget and not a statistic computed afterwards. Every scored variant spends a trial: a changed window, a flipped sign, a retried prompt. Autonomous Quant Researcher, a bounded research loop in which a language model proposes and trusted code decides, hashes its trial budget into a campaign contract before the first proposal is evaluated. It refuses duplicates and near-duplicates before they spend a trial, and it states that the count reported with a result is the number actually tried [3].

A rule can also follow the count as it grows. SEADS, the system presented in a methodological paper on agentic asset pricing, is an example. The tt-statistic of a candidate's rank IC must be at least the larger of 3.0 and a Bonferroni quantile computed from the cumulative number of candidates so far, under a rule set before the run [6].

05 /Scoring the search itself on the purged folds

A formulaic candidate has no fitted parameter, so its average IC over the KK test folds is its sample average: the folds change nothing for one candidate. They matter when something is fitted on the training side, and in a search the fitted object is the choice of candidate. For each fold the program picks the candidate with the highest mean IC on the purged and embargoed training dates Tk\mathcal{T}_k and records the IC of that candidate on the test block only.

Applied to the full sample, the same rule picks the candidate with the highest mean rank IC over all 240 dates, and that mean is 0.082. The fold-by-fold selection among the same 100 candidates earns a mean test IC of 0.006. Under the null of this simulation the second figure has expectation exactly zero, because after the purge the test labels share no return with anything the selection used. The value in one draw is noise around that zero, not small in a panel of 10 assets and 240 dates, and another seed can put it on either side. The program asserts only that the out-of-fold figure is below the in-sample one.

The validation theory cited above recommends that tuning be nested. An inner layer selects, an outer and strictly causal segment that no selection has touched confirms, and reported metrics are corrected for multiplicity [4]. The selection here is cross-validated, not nested in that sense: it has no such outer segment, and training dates lie on both sides of a test block.

The paper on agentic asset pricing raises the same point to the level of systems. Existing practice, it observes, backtests the factors or trades that a discovery system outputs and not the system that produced them. Its proposal is to re-execute the whole loop at a sequence of historical decision dates [6].

06 /From rank IC to weights, turnover and cost

A rank IC is a correlation, not a return. To trade a ranking it must become weights. Take ranks qt,iq_{t,i} from 00 to A−1A-1. The linear rank portfolio wt∗w^{*}_t below is dollar neutral and has unit gross exposure for even AA, because the centered ranks sum to zero and their absolute values sum to A2/4A^2/4. Positions are rebalanced every hh dates, so holding periods do not overlap.

Dollar-neutral rank weights with unit grossEQ. 6.1
wt,i∗=qt,i−(A−1)/2A2/4,∑iwt,i∗=0,∑i∣wt,i∗∣=1w^{*}_{t,i} = \frac{q_{t,i} - (A-1)/2}{A^{2}/4}, \qquad \sum_{i} w^{*}_{t,i} = 0, \qquad \sum_{i} \big|w^{*}_{t,i}\big| = 1

qt,iq_{t,i} is the rank of asset ii at date tt, from 00 to A−1A-1, without ties. The two sums hold for even AA and are asserted by the program on its first weight vector.

Let wt−w_{t^-} be the holdings before the rebalance at tt. One-way turnover τt\tau_t is half the absolute change in weights, since purchases equal sales when both portfolios are dollar neutral. With a cost κ\kappa per unit of notional traded, the rebalance costs 2κ τt2\kappa\,\tau_t in the units of the weights, and the net return of the period subtracts it from the gross return. A study that places a frozen statistical referee around factor-mining agents uses the same convention. It charges twice the per-side cost times one-way turnover each day and caps the one-way turnover of its book at 25 percent per day [7].

One-way turnover and net period returnEQ. 6.2
τt=12∑i∣wt,i−wt−,i∣,Rtnet=∑iwt,i yt,i−2κ τt\tau_t = \tfrac{1}{2}\sum_{i} \big| w_{t,i} - w_{t^-,i} \big|, \qquad R^{\mathrm{net}}_t = \sum_{i} w_{t,i}\, y_{t,i} - 2\kappa\,\tau_t

wt−w_{t^-} are the holdings before the rebalance and wtw_t after it, κ\kappa is the cost per unit of notional traded, and yt,iy_{t,i} is the return of asset ii over the hh dates that follow tt.

A turnover constraint fits the same arithmetic. Trading only a fraction λt\lambda_t of the way to the target caps one-way turnover at τˉ\bar\tau and keeps the portfolio dollar neutral by linearity. By the triangle inequality the step cannot raise gross exposure above one. It can leave it below one, and the program does not rescale the book.

Partial rebalance under a turnover capEQ. 6.3
wt=wt−+λt (wt∗−wt−),λt=min⁡ ⁣(1, τˉτt∗),τt∗=12∑i∣wt,i∗−wt−,i∣w_t = w_{t^-} + \lambda_t\,\big(w^{*}_t - w_{t^-}\big), \qquad \lambda_t = \min\!\Big(1,\ \frac{\bar\tau}{\tau^{*}_t}\Big), \qquad \tau^{*}_t = \tfrac{1}{2}\sum_{i} \big| w^{*}_{t,i} - w_{t^-,i} \big|

τt∗\tau^{*}_t is the turnover that a full rebalance to the target wt∗w^{*}_t would need and τˉ\bar\tau is the cap. The realized turnover is λtτt∗\lambda_t \tau^{*}_t, which never exceeds τˉ\bar\tau. The program measures turnover from the weights actually held.

The program applies both rules to the noise candidate with the highest zz, with a return volatility of 2 percent per date, scores whose autocorrelation is 0.9 per date and κ=10\kappa = 10 basis points. The opening trade is not charged, and the averages run over the rebalances that follow it. Full rebalancing has a mean one-way turnover of 0.43 per period, an in-sample gross return of 59.3 basis points per period and a net return of 50.7. With τˉ=0.2\bar\tau = 0.2 mean turnover is 0.20, and the figures are 40.5 gross and 36.5 net.

The capped book is not fully invested. Its mean gross exposure is 0.73, so part of the fall in its figures comes from smaller positions, which this cap produces because the book is not rescaled. All four return figures are artifacts of selection on a factor that is noise by construction. A filter that requires a positive net return at 10 basis points would keep this candidate, so in this panel a cost filter does not remove the false discovery.

07 /A reproducible program for the worked example

The program draws independent Gaussian returns for 10 assets, builds labels over h=5h = 5 dates and generates 100 candidates as autoregressive noise scores stored as cross-sectional ranks. The function named train implements the set Tk\mathcal{T}_k, and the function named shared counts the overlapping pairs. The function named trade implements the partial rebalance and measures turnover from the weights actually held; a cap of 1 stands for no cap. Six assertions, listed below, tie the printed numbers to the claims made above.

The seed is the edition date written as an integer. It was set before the first run and was not tuned. The two fold assertions do not depend on the seed. The other four include statements about a random draw and can fail for another seed without any error in the code. The program uses only the Python standard library and prints the same output on every run.

  • Statistic: reversed ranks give a rank correlation of −1-1, and the standard deviation of zz is within 0.2 of one on spaced dates and above 1.2 on overlapping dates.
  • Folds: without the embargo, no shared pair after the purge, some when the purge is one date short, and (K−1) h (h−1)(K-1)\,h\,(h-1) with no purge.
  • Embargo: for the second fold it removes exactly ee further training dates.
  • Tests: the best zz lies between the two thresholds, the clocked field stays below the Bonferroni one in absolute value, the event-dated field exceeds it, and out-of-fold IC is below in-sample IC.
  • Weights: the first weight vector sums to zero with unit gross exposure, and the capped book holds less mean gross exposure than the fully rebalanced one.
  • Trading: the largest capped turnover equals the cap, mean uncapped turnover exceeds it, and the uncapped gross return exceeds its cost.
python59 LINES EXECUTED · PYTHON 3.12.3 · 6 ASSERTS PASSED
# Synthetic unit-check fixture; illustrative only, no live market data or performance claims.
import random
from statistics import NormalDist, fmean, pstdev
random.seed(20261113)
A, T, H, N, K, E, L = 10, 240, 5, 100, 4, 5, 2  # assets dates horizon trials folds embargo lag
RHO, ALPHA, VOL, COST, CAP = 0.9, 0.05, 0.02, 0.001, 0.2
def ranks(v):
    order = sorted(v)
    return [order.index(x) for x in v]
def ic(a, b):  # Spearman rank correlation
    return 1 - 6 * sum((p - q) ** 2 for p, q in zip(a, b)) / (A ** 3 - A)
def panel(x):  # a candidate: persistent noise ranks
    return [ranks(x := [RHO * v + random.gauss(0, 1) for v in x]) for t in range(T)]
def train(k, purge, emb):  # training dates for test fold k
    return [s for s in range(T) if s < fold[k][0] - purge or s > fold[k][-1] + purge + emb]
def shared(purge, emb):  # train/test date pairs with a shared return
    return sum(abs(s - t) < H for k in range(K) for s in train(k, purge, emb) for t in fold[k])
def trade(cap):  # partial rebalance; turnover as traded
    hold = [w[0]]
    for v in w[1:]:
        gap = sum(abs(p - q) for p, q in zip(v, hold[-1])) / 2
        hold.append([q + min(1, cap / (gap or 1)) * (p - q) for p, q in zip(v, hold[-1])])
    turn = [sum(abs(p - q) for p, q in zip(u, v)) / 2 for u, v in zip(hold[1:], hold)]
    pay = [sum(p * q for p, q in zip(v, lab[t])) for v, t in zip(hold[1:], spaced[1:])]
    return [fmean(turn), max(turn), 1e4 * fmean(pay), fmean(sum(map(abs, v)) for v in hold[1:])]
fold, spaced = [range(k * T // K, (k + 1) * T // K) for k in range(K)], range(0, T, H)
ret = [[random.gauss(0, VOL) for i in range(A)] for t in range(T + H + L)]
lab = [[sum(ret[t + L + d][i] for d in range(1, H + 1)) for i in range(A)] for t in range(T)]
ry, unit = [ranks(v) for v in lab], ((A - 1) * H / T) ** 0.5
cand = [panel([random.gauss(0, (1 - RHO ** 2) ** -0.5) for i in range(A)]) for j in range(N)]
ics = [[ic(c[t], ry[t]) for t in range(T)] for c in cand]
z = [unit * sum(row[::H]) for row in ics]
z_all = [((A - 1) / T) ** 0.5 * sum(row) for row in ics]
naive, bonf = [NormalDist().inv_cdf(1 - a) for a in (ALPHA, ALPHA / N)]
field = [ranks([sum(ret[t + L - d][i] for d in range(L)) for i in range(A)]) for t in range(T + L)]
z_clock, z_event = [unit * sum(ic(field[t + lag], ry[t]) for t in spaced) for lag in (0, L)]
picks = [max(range(N), key=lambda j: sum(ics[j][s] for s in train(k, H - 1, E))) for k in range(K)]
oof, top = fmean(ics[picks[k]][s] for k in range(K) for s in fold[k]), max(map(fmean, ics))
w = [[(q - (A - 1) / 2) / (A * A / 4) for q in cand[z.index(max(z))][t]] for t in spaced]
full, capped = trade(1), trade(CAP)
assert ic(range(A), range(A)[::-1]) == -1 and abs(pstdev(z) - 1) < 0.2 < pstdev(z_all) - 1
assert shared(H - 1, 0) == 0 < shared(H - 2, 0) and shared(0, 0) == (K - 1) * H * (H - 1)
assert len(train(1, H - 1, 0)) - len(train(1, H - 1, E)) == E
assert naive < max(z) < bonf and abs(z_clock) < bonf < z_event and oof < top
assert abs(sum(map(abs, w[0])) - 1) + abs(sum(w[0])) < 1e-12 and capped[3] < full[3]
assert abs(capped[1] - CAP) < 1e-12 < full[0] - CAP and 2e4 * COST * full[0] < full[2]
out = {"z_on_clock": z_clock, "z_event_dated": z_event, "spaced_dates": len(spaced),
    "shared_pairs_raw": shared(0, 0), "shared_pairs_purged": shared(H - 1, E),
    "sd_z_spaced": pstdev(z), "sd_z_overlapping": pstdev(z_all), "naive_threshold": naive,
    "bonferroni_threshold": bonf, "best_z": max(z), "passes_naive": sum(v > naive for v in z),
    "passes_bonferroni": sum(v > bonf for v in z),
    "passes_naive_overlapping": sum(v > naive for v in z_all), "in_sample_mean_ic": top,
    "out_of_fold_mean_ic": oof, "turnover": full[0], "gross_bps": full[2],
    "net_bps": full[2] - 2e4 * COST * full[0], "turnover_capped": capped[0],
    "gross_bps_capped": capped[2], "net_bps_capped": capped[2] - 2e4 * COST * capped[0],
    "exposure_capped": capped[3]}
for key in out:
    print(f"RESULT {key} = {round(out[key], 4)}")
print("OK")
CAPTURED OUTPUT23 LINES · 6 ASSERT STATEMENTS, 6 EVALUATIONS PASSED · RUN NOV 14, 2026 · SHA-256 c77ed612820b…
RESULT z_on_clock = -1.3804
RESULT z_event_dated = 9.8045
RESULT spaced_dates = 48
RESULT shared_pairs_raw = 60
RESULT shared_pairs_purged = 0
RESULT sd_z_spaced = 0.9206
RESULT sd_z_overlapping = 1.6034
RESULT naive_threshold = 1.6449
RESULT bonferroni_threshold = 3.2905
RESULT best_z = 2.4511
RESULT passes_naive = 6
RESULT passes_bonferroni = 0
RESULT passes_naive_overlapping = 21
RESULT in_sample_mean_ic = 0.0822
RESULT out_of_fold_mean_ic = 0.0056
RESULT turnover = 0.4306
RESULT gross_bps = 59.3149
RESULT net_bps = 50.7021
RESULT turnover_capped = 0.2
RESULT gross_bps_capped = 40.5435
RESULT net_bps_capped = 36.5435
RESULT exposure_capped = 0.7293
OK

Standard output of the program above, captured when it was run in the journal's sandbox on Nov 14, 2026. The SHA-256 of the program text shown here equals the hash stored with that run. All inputs are synthetic.

08 /What recent audits and benchmarks report

An audit of a preserved language-model mining project finds that its historical comparison mixed metric populations: a mean over kept candidates was set against a mean of per-round maxima. A released catalog of 1,837 expressions excluded non-positive out-of-sample outcomes, which makes its zero-reversal rate non-informative [9]. A table of survivors has discarded the trial count.

Hubble's limitations section states the limits of its evidence: one temporal split and no full walk-forward protocol, an out-of-sample window of only 195 trading days, and no transaction-cost evaluation [1]. FactorBench, a benchmark of automated factor mining, uses 10 basis points per dollar traded as its default and finds long-short profitability fragile, with most methods near or below zero as costs increase [8].

The frozen-referee study names the limit of a count fixed in advance. A batch correction assumes a batch, and an agent that reads each verdict before it chooses its next proposal introduces optional stopping. Its referee therefore scores a candidate only on market outcomes revealed after submission. The price is a delay: in that paper's experiments the median wait of an admitted true factor is about 500 trading days [7].

09 /Assumptions and limitations of this demonstration

The statistic zz has an exact null variance only under the permutation null used here, and the program checks that scaling only to a tolerance of 0.2. A real candidate with no information about idiosyncratic returns can still load on a common factor. By a standard argument the variance of its IC then exceeds 1/(A−1)1/(A-1) unless scores or returns are neutralized, a case the simulation does not contain. The normal shape of zz is an approximation from 48 dates, and the Bonferroni tail level α/N\alpha/N leans on it far from the center. With 10 assets and 240 dates the stricter bar also leaves little power against a weak true factor.

Returns are independent across dates in the simulation, so the purge alone removes every shared return and the embargo guards against nothing here. Its length in a market with volatility clustering has to come from measured dependence, which the program does not model. The program also does not rerun the selection without the purge. The fall from 0.082 to 0.006 therefore shows what out-of-fold scoring does as a whole, and what the program measures of the purge itself is the count of shared pairs.

The budget argument needs every scored variant to be recorded, and it treats near-duplicates as full trials, which is conservative. The clock in the program covers data only. Hubble's authors note that temporal isolation at the data layer cannot stop a pretrained model from having absorbed findings published after the in-sample window begins [1].

The cost model is linear in traded notional, with no market impact, no borrow cost and no drift of weights between rebalances, and returns add across dates. The cost result belongs to this panel. The gross return of a selected noise candidate scales with return volatility and with the sampling error of the IC, which shrinks as the cross-section widens, and the cost does not. In a wider or calmer panel the same 10 basis points can exceed it. None of the simulated numbers is evidence about a real factor, a real system or a tradable strategy. They show how four separate accounts, of time, of overlap, of trials and of trading, change what the same noise appears to say.

∎

Research is complete only when the assumptions, experiment, and limitations can be reconstructed.

This perspective is research commentary. It does not constitute investment advice or evidence of live strategy performance. Experimental methods require project-specific testing. Equations and code illustrate research formulations, not measured live returns.

More perspectivesExplore the engineering atlas