Cross-sectional ranking test: Kronos-small on 182 US equities, 85 post-cutoff dates — IC +0.022 (t=1.39), fails after costs

[yonaoh10]

Summary

I ran a cross-sectional ranking test of zero-shot Kronos-small on 182 US equities across 85 rebalance dates, entirely after the stated June-2024 pretraining cutoff, using the paper's own daily and inference settings.

Result: IC = +0.0220, Newey-West t = 1.39 (95% CI −0.009 to +0.053). Consistently positive across all four sub-periods, but not statistically significant, and it does not survive transaction costs. Notably, it is not merely a repackaging of standard factors — see the orthogonalization section.

I am not claiming Kronos is broken, and this single test is underpowered (see Limitations). I am posting because every prior evaluation I could find in this repo tests single-asset forecasting, whereas the paper's Investment Simulation ranks stocks cross-sectionally — so this seemed like a gap worth filling.

Why this test differs from the existing reports

Reports such as #354 and #355 measure per-asset forecast error and directional accuracy. But the paper's Investment Simulation "construct[s] portfolios with the top-k stocks ranked by each model's predictive signals," and finetune/qlib_test.py:274-279 builds signal = predicted_close − last_close and ranks across instruments. A model can be useless at absolute price prediction and still rank assets usefully. That is what I tested.

Setup

• Model: NeoQuasar/Kronos-small + NeoQuasar/Kronos-Tokenizer-base, .eval(), CPU, fp32
• Universe: 182 US equities. Includes 64 names that fell hard or were removed from indices (SEDG, ENPH, MRNA, PYPL, DOCU, VFC, ZM, NCLH, WBD …), to limit survivorship bias. The 118 "surviving" large caps returned +143% median over the sample; the 64 added names returned +7% median.
• Data: daily OHLCV, split/dividend adjusted (yfinance auto_adjust=True)
• Period: 2024-07-01 → 2026-07-07 only — strictly after the June-2024 pretraining cutoff stated in Appendix D
• Settings, matching the paper: lookback = 40, horizon = 12 (Table 8, Daily); T = 0.6, top_p = 0.90, N = 10 samples (Table 6, Investment Simulation)
• 85 rebalance dates, every 6 trading days
• No look-ahead: signal uses bars through day t−1; entry at the close of day t; exit at the close of day t+12. Normalization statistics come only from the lookback window.
• Signal: predicted_close[last] / close[t−1] − 1, ranked cross-sectionally

Results

signal	mean IC	t (Newey-West)	% dates positive
Kronos (last)	+0.0220	+1.39	54%
Kronos (mean of path)	+0.0181	+1.13	52%
5-day reversal	+0.0316	+1.97	55%
12-1 momentum	+0.0132	+0.67	56%
21-day momentum	−0.0256	−1.13	47%
low volatility (90d)	−0.0081	−0.24	48%

Sub-period ICs: 2024 H2 +0.0095 · 2025 H1 +0.0305 · 2025 H2 +0.0348 · 2026 H1 +0.0144 — positive in all four.

Orthogonalization against standard factors (the interesting part)

I regressed the Kronos cross-section on ranks of {21-day momentum, 12-1 momentum, 5-day reversal, 90-day volatility} each date, and recomputed IC on the residual:

• Mean R² vs. those factors: 0.316
• Raw IC +0.0220 (t = 1.39) → residual IC +0.0134 (t = 1.33) — retaining 96% of the t-stat

So roughly two-thirds of the signal is not explained by these standard factors. Kronos appears to carry genuinely independent cross-sectional information. It is simply too weak, at this model size and in this sample, to clear a significance bar or pay for trading costs.

Cost and concentration

Long-short decile portfolio (top 18 vs bottom 18, equal weight):

	annualized	Sharpe	t
gross	+15.56%	0.77	1.51
net of 10 bp/side	+7.16%	0.35	0.69

The concentration is the bigger problem. Cumulative gross P&L was +63.0%, of which:

• best single date: +14.7% (23% of all profit)
• best 5 of 85 dates: +57.4% (91% of all profit)
• remaining 80 dates: +5.5% combined

Excluding the best 5 dates, the net strategy is negative. Deciles are also non-monotonic (D1 +0.30%, D6 +0.19%, D9 +0.86%, D10 +1.07%).

A separate, actionable finding: the sampled spread is far too narrow

Independent of ranking, I measured whether the sampled path distribution is calibrated, since sample_count paths are a natural source of uncertainty bands.

Generating 1000 independent paths (batch-tiled with sample_count=1, which samples independently per batch row) on 5-min data:

• Across 8 windows, the nominal 90% band covered 58% of realized outcomes; the 50% band covered 22%.
• Against volatility-matched benchmarks (same instrument, same clock slot), the spread is ≈2.7× too narrow (day-block bootstrap 95% CI 2.0–3.4).
• Setting top_p = 1.0 widened the band (1.54% → 1.89%) but did not improve coverage (58.3% → 57.8%), so this is not a nucleus-sampling artifact.
• The tokenizer is not the cause: encode→decode round-trip on known bars preserves per-bar volatility at 0.95× and 24-bar spread at 0.99×. The compression is faithful; the shortfall arises in the autoregressive rollout (roughly half damped per-step amplitude, half excess mean reversion — increment lag-1 autocorrelation −0.25 vs −0.12 in matched real data).

Practical implication: the spread across sampled paths should not be used as a risk measure — realized moves are materially larger than it implies. It may be worth documenting this in the README.

Minor code note

RotaryPositionalEmbedding caches cos_cached / sin_cached as plain attributes rather than registered buffers (model/module.py:289-301), and _update_cos_sin_cache only recomputes when seq_len changes. So .to(device) does not move the cache and it is never invalidated on a device change. Reusing one Kronos/KronosTokenizer object across two KronosPredictors with different devices raises:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, mps:0 and cpu!
  model/module.py:306 in forward

Registering them as non-persistent buffers, or keying the cache on (seq_len, device, dtype), would fix it.

Limitations — please weigh these

1. Underpowered. IC std across dates was 0.142, so at the observed effect size roughly 166 dates (~8 years) would be needed for t = 2. I have 85. The correct reading is "not demonstrated," not "demonstrated absent." The 95% CI includes both zero and a tradeable +0.05.
2. Kronos-small only. The paper shows performance scaling with model size; Kronos-base may do better. I did not have the compute (CPU only) to repeat at base scale.
3. Zero-shot, no fine-tuning. I tested Kronos-mini on split-adjusted, regular-session 30-minute US OHLCV from eight liquid symbols #355 suggests fine-tuning improves token loss without improving trading outcomes, but that was a different setup.
4. US equities only; the paper's Investment Simulation used Chinese A-shares.
5. Survivorship bias is reduced but not eliminated — a point-in-time index membership file would be better, and some delisted tickers (WBA, PARA, SQ, CTLT, IPG) cannot be retrieved from yfinance at all.
6. A caution for anyone replicating: I initially got a permutation-test p = 0.008 alongside t = 1.39. Permuting the signal within each date produces a null whose per-date IC std is only ~1/√N_stocks (0.074 here), while the real IC series varied at 0.142 because true IC shifts with regime. The permutation test was anti-conservative by ~2×. Trust the t-stat computed from the observed IC series.

Questions

1. Is daily-frequency US equity cross-sectional ranking an intended zero-shot use case, or is fine-tuning per market expected?
2. For the Investment Simulation, was the signal computed in normalized (per-instrument σ) space, as finetune/qlib_test.py does, or in price/return space? That choice is an implicit volatility scaling and may matter materially for ranking.
3. What lookback and horizon were used for the Investment Simulation specifically? Table 8 gives 40/12 for the daily forecasting task, while finetune/config.py uses 90/10.
4. Is the narrow sampled spread expected behavior, and is there a recommended way to obtain calibrated intervals?

Happy to share the full harness and the raw prediction arrays if useful. Thanks for open-sourcing the model and the fine-tuning pipeline — the setup was straightforward to work with.