Reproducibility report: Kronos-mini underperforms a persistence baseline on AAPL daily OHLCV across 1,800 rolling forecasts

[petrk94]

Summary

I tested zero-shot Kronos-mini forecasting on 1,000 daily AAPL OHLCV rows using a rolling evaluation with 100 historical forecast origins, two lookback lengths, three forecast horizons, and three stochastic repeats per configuration (1,800 forecasts in total).

In this setup, every tested Kronos configuration had a higher mean close-price MAPE than a simple persistence baseline that predicts the last observed close for every future step. Directional accuracy ranged from approximately 48.5% to 54.3%.

I am not claiming that Kronos is generally broken. I am opening this report to ask whether the evaluation setup is valid for the released pretrained model, whether daily US equities are an intended zero-shot use case, and whether additional preprocessing or fine-tuning is expected.

Model and data

• Model: NeoQuasar/Kronos-mini
• Tokenizer: NeoQuasar/Kronos-Tokenizer-2k
• Device: CPU
• Asset: AAPL
• Frequency: daily
• Input columns: timestamps, open, high, low, close, volume
• Dataset size: 1,000 rows
• Evaluated forecast origins: 100
• Forecast-origin spacing: 5 trading rows
• Target period covered by the rolling evaluation: 2024-07-26 through 2026-07-24
• Lookbacks: 250 and 280
• Horizons: 2, 3, and 5 trading rows
• Repeats per window/configuration: 3
• Temperature: 0.8
• top_p: 0.9
• sample_count: 3
• Total forecasts: 1,800

The original run did not set an explicit random seed. Therefore, individual sampled paths are not expected to reproduce exactly, but the aggregate result should be testable. The attached direct reproduction script supports a deterministic --seed-base option for follow-up checks.

Baseline and metrics

The persistence baseline predicts:

predicted_close[t + k] = last_observed_close

Metrics:

• Mean absolute percentage error (MAPE) on close
• MAE and RMSE on close
• Percentage of individual runs in which Kronos beats persistence
• Step-by-step directional accuracy, starting from the last observed close
• Endpoint directional accuracy

Results

Lookback	Horizon	Kronos mean MAPE	Persistence mean MAPE	Relative MAPE vs persistence	Runs beating persistence	Direction accuracy	Endpoint direction hit	Worst MAPE
250	2	1.6493%	1.3771%	19.77% worse	37.33%	54.33%	54.67%	7.6907%
280	2	1.6754%	1.3771%	21.66% worse	33.67%	50.33%	54.33%	7.7409%
250	3	1.8220%	1.6239%	12.20% worse	40.67%	54.11%	54.33%	10.6218%
280	3	1.8423%	1.6239%	13.45% worse	41.00%	52.44%	57.00%	8.4011%
250	5	2.4688%	2.1702%	13.76% worse	36.33%	48.53%	47.00%	13.1418%
280	5	2.4481%	2.1702%	12.81% worse	38.67%	49.20%	50.33%	13.3798%

All 1,800 runs completed successfully.

Reproduction

1. Clone the repository and record the commit

git clone https://github.com/shiyu-coder/Kronos.git
cd Kronos
git rev-parse HEAD

The tested repository revision was:

67b630e67f6a18c9e9be918d9b4337c960db1e9a

2. Create the environment

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt

For the optional Yahoo Finance downloader only:

pip install yfinance

3. Provide the dataset

For exact reproduction, place the attached frozen dataset at:

data/AAPL_1d_1000.csv

Its columns must be:

timestamps,open,high,low,close,volume

Because historical vendor data can be revised, attaching the exact CSV is preferable to redownloading it. An optional download_aapl_1000.py convenience script is included, but it may not recreate byte-identical historical values later.

4. Copy the direct reproduction script to the repository root

reproduce_aapl_daily_benchmark.py

This script imports Kronos, KronosTokenizer, and KronosPredictor directly. It does not depend on a modified Web UI or HTTP API.

5. Optional smoke test

python reproduce_aapl_daily_benchmark.py \
  --csv data/AAPL_1d_1000.csv \
  --device cpu \
  --windows 3 \
  --repeats 1 \
  --output-dir smoke_test

6. Full test

python reproduce_aapl_daily_benchmark.py \
  --csv data/AAPL_1d_1000.csv \
  --device cpu \
  --lookbacks 250 280 \
  --horizons 2 3 5 \
  --windows 100 \
  --step 5 \
  --repeats 3 \
  --temperature 0.8 \
  --top-p 0.9 \
  --sample-count 3 \
  --output-dir reproduction_results

For a deterministic follow-up run, add for example:

--seed-base 12345

The script writes:

reproduction_results/runs.csv
reproduction_results/summary.csv
reproduction_results/environment.json

Tested environment

• Repository commit: 67b630e67f6a18c9e9be918d9b4337c960db1e9a
• Python: 3.10.12
• PyTorch: 2.13.0+cpu
• pandas: 2.2.2
• NumPy: 2.2.6
• CUDA available: False
• Device: CPU only
• OS: Zorin OS / Ubuntu-based Linux
• Kernel: Linux 6.8.0-94-generic #96~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 16 13:19:05 UTC 2 x86_64
• CPU: AMD Ryzen 3 7330U, 4 cores / 8 threads
• RAM: 8 GB

Raw environment output:

=== COMMIT ===
67b630e67f6a18c9e9be918d9b4337c960db1e9a
=== PYTHON ===
Python 3.10.12
=== PACKAGES ===
torch: 2.13.0+cpu
pandas: 2.2.2
numpy: 2.2.6
CUDA available: False
=== SYSTEM ===
Linux petr-zorinos 6.8.0-94-generic #96~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 16 13:19:05 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Questions

1. Is zero-shot forecasting of daily US equity OHLCV an intended use case for the released pretrained models?
2. Is feeding raw OHLCV through KronosPredictor.predict() the expected preprocessing path for this use case, or should returns/log-prices/another transformation be used?
3. Is a persistence baseline the appropriate minimum benchmark for the released model on daily prices?
4. Are there recommended seeds, sampling parameters, or evaluation scripts for reproducible comparisons?
5. Is fine-tuning expected before evaluating daily equity data?
6. Are there known differences between the training distribution and US daily equities that could explain these results?

Attached files

• reproduce_aapl_daily_benchmark.py — direct reproduction script
• AAPL_1d_1000.csv — exact frozen input data
• results/config.json — original benchmark configuration
• results/summary.csv — aggregate results
• results/runs.csv — all 1,800 individual runs
• download_aapl_1000.py — optional convenience downloader

config.json
runs.csv
download_aapl_1000.py
summary.csv
reproduce_aapl_daily_benchmark.py

AAPL_1d_1000.csv

Thank you for taking a look. I would appreciate confirmation of whether this is a valid evaluation of the released zero-shot model, or guidance on the intended preprocessing/evaluation protocol.