In an AI Factor Search, the Number of Candidates Tried Is Part of the Result
On 100 simulated noise factors with no leakage, a correctly scaled test still leaves six apparent discoveries. A threshold set by the trial count removes all six. In this ten-asset panel a cost filter keeps the best one.
| Section | Sources cited | Equations | Code listings |
|---|---|---|---|
| 1. What a search over one hundred noise factors shows | 1 | 0 | 0 |
| 2. Availability clocks and the labels they protect | 2 | 2 | 0 |
| 3. Purged and embargoed folds for overlapping labels | 4, 5 | 2 | 0 |
| 4. The trial count as part of the test | 3, 6 | 1 | 0 |
| 5. Scoring the search itself on the purged folds | 4, 6 | 0 | 0 |
| 6. From rank IC to weights, turnover and cost | 7 | 3 | 0 |
| 7. A reproducible program for the worked example | none | 0 | 1 |
| 8. What recent audits and benchmarks report | 1, 7, 8, 9 | 0 | 0 |
| 9. Assumptions and limitations of this demonstration | 1 | 0 | 0 |
01 /What a search over one hundred noise factors shows
A language model can propose formulaic factors in bulk, and each proposal is one more draw from the distribution of lucky backtests. This article shows, on a simulated market in which no candidate carries any information, that the number of candidates tried belongs in the result itself. No leak is present, the labels do not overlap and the statistic is scaled by its exact null variance. Still, 6 of 100 noise factors pass a one-sided test at the 5 percent level. A threshold computed from the declared budget of 100 trials passes none. Scored out of fold, the search that shows a rank IC of 0.082 in sample earns a figure whose expectation is zero, 0.006 in this draw.
Published systems already report the ingredients. Hubble, an agentic factor-mining framework, describes a main run that evaluates 104 valid candidates across three rounds, after which the top five factors are fixed and validated on a held-out period [1]. That description contains a count of attempts and a selection step, and the count is the quantity this article follows.
Four controls are kept apart because each answers a different error. The availability clock decides what a feature may contain at a decision date. Purging and an embargo decide which training dates may sit next to a test fold. The trial budget sets the significance bar. Turnover and cost decide whether a rank correlation says anything about money. One Python program of 59 lines computes every simulation result quoted below.
02 /Availability clocks and the labels they protect
Index assets by and decision dates by , and write for the return of asset realized over date . A raw field carries two timestamps. Its event date says which period the value describes. Its availability time says when the value could first have been read. A feature used at date may contain only values with , so it is built from the latest admissible event date. The label is the return over the next dates and never touches a return dated or earlier.
is the raw value of the field for event date and asset , and is its availability time. In the program with a publication lag of dates.
The label sums the next one-date returns. is the Spearman rank correlation, taken across the assets, between feature and label at date .
The program contains one field that makes the difference measurable. Its value for event date is the sum of the returns that follow , as for a value stamped with its period but finalized only at publication, so . On the availability clock the feature at is , a sum of returns already realized, which says nothing about independent future returns. Read by event date, the feature at is , a piece of the label. On the statistic of the next section the first version scores and the second .
The second score exceeds every threshold used in this article. It is not a multiple-testing problem, and no threshold repairs it. The Availability Contract, a recent proposal for earnings-call benchmarks, observes that chronological train and test splits do not prevent a timing mismatch that sits inside a single event. It requires a benchmark to state the input boundary, the evidence of input availability and the start of the outcome window [2].
03 /Purged and embargoed folds for overlapping labels
With the labels of neighboring dates share returns: the label windows of dates and intersect exactly when . Split the dates into contiguous test blocks, and let block run from date to date . A training date within dates of the block, on either side, has a label that shares a return with a test label. Purging deletes those dates. The embargo deletes further dates after the block, as a margin against serial dependence that outlasts the label window.
and are the first and last dates of test block , is the label horizon and the embargo. Every in the set satisfies for every test date , so no label window is shared.
The program counts the pairs of one training date and one test date whose label windows share a return. With , and the unpurged split has 60 such pairs, which equals the closed form . Each interior boundary is crossed from both sides, and pairs cross it at distance for . After the purge the count is 0, with or without the embargo of dates, and it is positive again when the purge is one date short.
A recent theory of time-series validation gives the same accounting. It prices purged -fold with an embargo at a cleared zone of one label horizon before each test block and one horizon plus the embargo after it [4]. In the date arithmetic used here that is dates on each side and more after. A forward-chaining splitter needs only the first half. In scikit-learn, TimeSeriesSplit trains on the folds that precede the test fold and offers a gap parameter that drops samples from the end of each training set [5].
Overlap also distorts the test statistic of a single candidate when its scores persist from date to date. Take as null hypothesis that a candidate's ranks are a random permutation independent of the labels. Each rank IC then has mean zero and variance , the classical null moments of Spearman's coefficient without ties. Dates spaced apart have disjoint label windows. When returns are independent across dates, as in the simulation, their ICs are independent given the scores, and the standardized mean over those dates has unit variance.
The summands use disjoint label windows. Under the null, with returns independent across dates, they are independent given the scores and each has variance . Then has unit variance and is approximately standard normal.
Across the 100 noise candidates the program measures a standard deviation of 0.92 for on the 48 spaced dates. The same scaling applied to all 240 overlapping dates, as if they were independent, gives a standard deviation of 1.60. There, 21 of the 100 candidates clear the one-sided 5 percent threshold where 5 are expected. The inflation needs both overlapping labels and persistent scores, here with an autocorrelation of 0.9 per date. Ranks redrawn independently at each date would give uncorrelated ICs whatever the labels share, a case that the null settles and the program does not run. With persistent scores, a pipeline that counts overlapping dates as independent manufactures significance before any search begins.
04 /The trial count as part of the test
Suppose candidates are scored and each is approximately standard normal when candidate is noise. A one-sided test at level passes a candidate when exceeds , and the expected number of noise candidates that pass is whatever their dependence. The probability that at least one of them exceeds a threshold is limited by the union bound, and solving that bound for level gives the Bonferroni threshold .
is the standard normal distribution function, the number of trials and the family-wise error level. For standard normal the inequality is the union bound over null candidates, which needs no independence. At its right side equals .
In the program and . The single-test threshold is 1.645 and the expected number of passes is . The draw produces 6, and the best candidate has , a discovery for any pipeline that forgets how many candidates it scored. The Bonferroni threshold is , and no candidate passes it. That outcome belongs to this seeded draw: the bound itself allows a noise candidate above in up to 5 percent of draws.
The bound holds for the that was fixed before the first result, so the count is a budget and not a statistic computed afterwards. Every scored variant spends a trial: a changed window, a flipped sign, a retried prompt. Autonomous Quant Researcher, a bounded research loop in which a language model proposes and trusted code decides, hashes its trial budget into a campaign contract before the first proposal is evaluated. It refuses duplicates and near-duplicates before they spend a trial, and it states that the count reported with a result is the number actually tried [3].
A rule can also follow the count as it grows. SEADS, the system presented in a methodological paper on agentic asset pricing, is an example. The -statistic of a candidate's rank IC must be at least the larger of 3.0 and a Bonferroni quantile computed from the cumulative number of candidates so far, under a rule set before the run [6].
05 /Scoring the search itself on the purged folds
A formulaic candidate has no fitted parameter, so its average IC over the test folds is its sample average: the folds change nothing for one candidate. They matter when something is fitted on the training side, and in a search the fitted object is the choice of candidate. For each fold the program picks the candidate with the highest mean IC on the purged and embargoed training dates and records the IC of that candidate on the test block only.
Applied to the full sample, the same rule picks the candidate with the highest mean rank IC over all 240 dates, and that mean is 0.082. The fold-by-fold selection among the same 100 candidates earns a mean test IC of 0.006. Under the null of this simulation the second figure has expectation exactly zero, because after the purge the test labels share no return with anything the selection used. The value in one draw is noise around that zero, not small in a panel of 10 assets and 240 dates, and another seed can put it on either side. The program asserts only that the out-of-fold figure is below the in-sample one.
The validation theory cited above recommends that tuning be nested. An inner layer selects, an outer and strictly causal segment that no selection has touched confirms, and reported metrics are corrected for multiplicity [4]. The selection here is cross-validated, not nested in that sense: it has no such outer segment, and training dates lie on both sides of a test block.
The paper on agentic asset pricing raises the same point to the level of systems. Existing practice, it observes, backtests the factors or trades that a discovery system outputs and not the system that produced them. Its proposal is to re-execute the whole loop at a sequence of historical decision dates [6].
06 /From rank IC to weights, turnover and cost
A rank IC is a correlation, not a return. To trade a ranking it must become weights. Take ranks from to . The linear rank portfolio below is dollar neutral and has unit gross exposure for even , because the centered ranks sum to zero and their absolute values sum to . Positions are rebalanced every dates, so holding periods do not overlap.
is the rank of asset at date , from to , without ties. The two sums hold for even and are asserted by the program on its first weight vector.
Let be the holdings before the rebalance at . One-way turnover is half the absolute change in weights, since purchases equal sales when both portfolios are dollar neutral. With a cost per unit of notional traded, the rebalance costs in the units of the weights, and the net return of the period subtracts it from the gross return. A study that places a frozen statistical referee around factor-mining agents uses the same convention. It charges twice the per-side cost times one-way turnover each day and caps the one-way turnover of its book at 25 percent per day [7].
are the holdings before the rebalance and after it, is the cost per unit of notional traded, and is the return of asset over the dates that follow .
A turnover constraint fits the same arithmetic. Trading only a fraction of the way to the target caps one-way turnover at and keeps the portfolio dollar neutral by linearity. By the triangle inequality the step cannot raise gross exposure above one. It can leave it below one, and the program does not rescale the book.
is the turnover that a full rebalance to the target would need and is the cap. The realized turnover is , which never exceeds . The program measures turnover from the weights actually held.
The program applies both rules to the noise candidate with the highest , with a return volatility of 2 percent per date, scores whose autocorrelation is 0.9 per date and basis points. The opening trade is not charged, and the averages run over the rebalances that follow it. Full rebalancing has a mean one-way turnover of 0.43 per period, an in-sample gross return of 59.3 basis points per period and a net return of 50.7. With mean turnover is 0.20, and the figures are 40.5 gross and 36.5 net.
The capped book is not fully invested. Its mean gross exposure is 0.73, so part of the fall in its figures comes from smaller positions, which this cap produces because the book is not rescaled. All four return figures are artifacts of selection on a factor that is noise by construction. A filter that requires a positive net return at 10 basis points would keep this candidate, so in this panel a cost filter does not remove the false discovery.
07 /A reproducible program for the worked example
The program draws independent Gaussian returns for 10 assets, builds labels over dates and generates 100 candidates as autoregressive noise scores stored as cross-sectional ranks. The function named train implements the set , and the function named shared counts the overlapping pairs. The function named trade implements the partial rebalance and measures turnover from the weights actually held; a cap of 1 stands for no cap. Six assertions, listed below, tie the printed numbers to the claims made above.
The seed is the edition date written as an integer. It was set before the first run and was not tuned. The two fold assertions do not depend on the seed. The other four include statements about a random draw and can fail for another seed without any error in the code. The program uses only the Python standard library and prints the same output on every run.
- Statistic: reversed ranks give a rank correlation of , and the standard deviation of is within 0.2 of one on spaced dates and above 1.2 on overlapping dates.
- Folds: without the embargo, no shared pair after the purge, some when the purge is one date short, and with no purge.
- Embargo: for the second fold it removes exactly further training dates.
- Tests: the best lies between the two thresholds, the clocked field stays below the Bonferroni one in absolute value, the event-dated field exceeds it, and out-of-fold IC is below in-sample IC.
- Weights: the first weight vector sums to zero with unit gross exposure, and the capped book holds less mean gross exposure than the fully rebalanced one.
- Trading: the largest capped turnover equals the cap, mean uncapped turnover exceeds it, and the uncapped gross return exceeds its cost.
# Synthetic unit-check fixture; illustrative only, no live market data or performance claims.
import random
from statistics import NormalDist, fmean, pstdev
random.seed(20261113)
A, T, H, N, K, E, L = 10, 240, 5, 100, 4, 5, 2 # assets dates horizon trials folds embargo lag
RHO, ALPHA, VOL, COST, CAP = 0.9, 0.05, 0.02, 0.001, 0.2
def ranks(v):
order = sorted(v)
return [order.index(x) for x in v]
def ic(a, b): # Spearman rank correlation
return 1 - 6 * sum((p - q) ** 2 for p, q in zip(a, b)) / (A ** 3 - A)
def panel(x): # a candidate: persistent noise ranks
return [ranks(x := [RHO * v + random.gauss(0, 1) for v in x]) for t in range(T)]
def train(k, purge, emb): # training dates for test fold k
return [s for s in range(T) if s < fold[k][0] - purge or s > fold[k][-1] + purge + emb]
def shared(purge, emb): # train/test date pairs with a shared return
return sum(abs(s - t) < H for k in range(K) for s in train(k, purge, emb) for t in fold[k])
def trade(cap): # partial rebalance; turnover as traded
hold = [w[0]]
for v in w[1:]:
gap = sum(abs(p - q) for p, q in zip(v, hold[-1])) / 2
hold.append([q + min(1, cap / (gap or 1)) * (p - q) for p, q in zip(v, hold[-1])])
turn = [sum(abs(p - q) for p, q in zip(u, v)) / 2 for u, v in zip(hold[1:], hold)]
pay = [sum(p * q for p, q in zip(v, lab[t])) for v, t in zip(hold[1:], spaced[1:])]
return [fmean(turn), max(turn), 1e4 * fmean(pay), fmean(sum(map(abs, v)) for v in hold[1:])]
fold, spaced = [range(k * T // K, (k + 1) * T // K) for k in range(K)], range(0, T, H)
ret = [[random.gauss(0, VOL) for i in range(A)] for t in range(T + H + L)]
lab = [[sum(ret[t + L + d][i] for d in range(1, H + 1)) for i in range(A)] for t in range(T)]
ry, unit = [ranks(v) for v in lab], ((A - 1) * H / T) ** 0.5
cand = [panel([random.gauss(0, (1 - RHO ** 2) ** -0.5) for i in range(A)]) for j in range(N)]
ics = [[ic(c[t], ry[t]) for t in range(T)] for c in cand]
z = [unit * sum(row[::H]) for row in ics]
z_all = [((A - 1) / T) ** 0.5 * sum(row) for row in ics]
naive, bonf = [NormalDist().inv_cdf(1 - a) for a in (ALPHA, ALPHA / N)]
field = [ranks([sum(ret[t + L - d][i] for d in range(L)) for i in range(A)]) for t in range(T + L)]
z_clock, z_event = [unit * sum(ic(field[t + lag], ry[t]) for t in spaced) for lag in (0, L)]
picks = [max(range(N), key=lambda j: sum(ics[j][s] for s in train(k, H - 1, E))) for k in range(K)]
oof, top = fmean(ics[picks[k]][s] for k in range(K) for s in fold[k]), max(map(fmean, ics))
w = [[(q - (A - 1) / 2) / (A * A / 4) for q in cand[z.index(max(z))][t]] for t in spaced]
full, capped = trade(1), trade(CAP)
assert ic(range(A), range(A)[::-1]) == -1 and abs(pstdev(z) - 1) < 0.2 < pstdev(z_all) - 1
assert shared(H - 1, 0) == 0 < shared(H - 2, 0) and shared(0, 0) == (K - 1) * H * (H - 1)
assert len(train(1, H - 1, 0)) - len(train(1, H - 1, E)) == E
assert naive < max(z) < bonf and abs(z_clock) < bonf < z_event and oof < top
assert abs(sum(map(abs, w[0])) - 1) + abs(sum(w[0])) < 1e-12 and capped[3] < full[3]
assert abs(capped[1] - CAP) < 1e-12 < full[0] - CAP and 2e4 * COST * full[0] < full[2]
out = {"z_on_clock": z_clock, "z_event_dated": z_event, "spaced_dates": len(spaced),
"shared_pairs_raw": shared(0, 0), "shared_pairs_purged": shared(H - 1, E),
"sd_z_spaced": pstdev(z), "sd_z_overlapping": pstdev(z_all), "naive_threshold": naive,
"bonferroni_threshold": bonf, "best_z": max(z), "passes_naive": sum(v > naive for v in z),
"passes_bonferroni": sum(v > bonf for v in z),
"passes_naive_overlapping": sum(v > naive for v in z_all), "in_sample_mean_ic": top,
"out_of_fold_mean_ic": oof, "turnover": full[0], "gross_bps": full[2],
"net_bps": full[2] - 2e4 * COST * full[0], "turnover_capped": capped[0],
"gross_bps_capped": capped[2], "net_bps_capped": capped[2] - 2e4 * COST * capped[0],
"exposure_capped": capped[3]}
for key in out:
print(f"RESULT {key} = {round(out[key], 4)}")
print("OK")CAPTURED OUTPUT23 LINES · 6 ASSERT STATEMENTS, 6 EVALUATIONS PASSED · RUN NOV 14, 2026 · SHA-256 c77ed612820b…
RESULT z_on_clock = -1.3804 RESULT z_event_dated = 9.8045 RESULT spaced_dates = 48 RESULT shared_pairs_raw = 60 RESULT shared_pairs_purged = 0 RESULT sd_z_spaced = 0.9206 RESULT sd_z_overlapping = 1.6034 RESULT naive_threshold = 1.6449 RESULT bonferroni_threshold = 3.2905 RESULT best_z = 2.4511 RESULT passes_naive = 6 RESULT passes_bonferroni = 0 RESULT passes_naive_overlapping = 21 RESULT in_sample_mean_ic = 0.0822 RESULT out_of_fold_mean_ic = 0.0056 RESULT turnover = 0.4306 RESULT gross_bps = 59.3149 RESULT net_bps = 50.7021 RESULT turnover_capped = 0.2 RESULT gross_bps_capped = 40.5435 RESULT net_bps_capped = 36.5435 RESULT exposure_capped = 0.7293 OK
Standard output of the program above, captured when it was run in the journal's sandbox on Nov 14, 2026. The SHA-256 of the program text shown here equals the hash stored with that run. All inputs are synthetic.
08 /What recent audits and benchmarks report
An audit of a preserved language-model mining project finds that its historical comparison mixed metric populations: a mean over kept candidates was set against a mean of per-round maxima. A released catalog of 1,837 expressions excluded non-positive out-of-sample outcomes, which makes its zero-reversal rate non-informative [9]. A table of survivors has discarded the trial count.
Hubble's limitations section states the limits of its evidence: one temporal split and no full walk-forward protocol, an out-of-sample window of only 195 trading days, and no transaction-cost evaluation [1]. FactorBench, a benchmark of automated factor mining, uses 10 basis points per dollar traded as its default and finds long-short profitability fragile, with most methods near or below zero as costs increase [8].
The frozen-referee study names the limit of a count fixed in advance. A batch correction assumes a batch, and an agent that reads each verdict before it chooses its next proposal introduces optional stopping. Its referee therefore scores a candidate only on market outcomes revealed after submission. The price is a delay: in that paper's experiments the median wait of an admitted true factor is about 500 trading days [7].
09 /Assumptions and limitations of this demonstration
The statistic has an exact null variance only under the permutation null used here, and the program checks that scaling only to a tolerance of 0.2. A real candidate with no information about idiosyncratic returns can still load on a common factor. By a standard argument the variance of its IC then exceeds unless scores or returns are neutralized, a case the simulation does not contain. The normal shape of is an approximation from 48 dates, and the Bonferroni tail level leans on it far from the center. With 10 assets and 240 dates the stricter bar also leaves little power against a weak true factor.
Returns are independent across dates in the simulation, so the purge alone removes every shared return and the embargo guards against nothing here. Its length in a market with volatility clustering has to come from measured dependence, which the program does not model. The program also does not rerun the selection without the purge. The fall from 0.082 to 0.006 therefore shows what out-of-fold scoring does as a whole, and what the program measures of the purge itself is the count of shared pairs.
The budget argument needs every scored variant to be recorded, and it treats near-duplicates as full trials, which is conservative. The clock in the program covers data only. Hubble's authors note that temporal isolation at the data layer cannot stop a pretrained model from having absorbed findings published after the in-sample window begins [1].
The cost model is linear in traded notional, with no market impact, no borrow cost and no drift of weights between rebalances, and returns add across dates. The cost result belongs to this panel. The gross return of a selected noise candidate scales with return volatility and with the sampling error of the IC, which shrinks as the cross-section widens, and the cost does not. In a wider or calmer panel the same 10 basis points can exceed it. None of the simulated numbers is evidence about a real factor, a real system or a tradable strategy. They show how four separate accounts, of time, of overlap, of trials and of trading, change what the same noise appears to say.
Research is complete only when the assumptions, experiment, and limitations can be reconstructed.
This perspective is research commentary. It does not constitute investment advice or evidence of live strategy performance. Experimental methods require project-specific testing. Equations and code illustrate research formulations, not measured live returns.