Skip to content

Latest commit

 

History

History
1046 lines (819 loc) · 30.9 KB

03_batch_processing.md

File metadata and controls

1046 lines (819 loc) · 30.9 KB

Batch Processing Tutorial

Comprehensive guide to calculating multiple technical indicators efficiently with GPU acceleration.


Why Batch Processing Matters

Batch processing is THE critical optimization in kimsfinance. When calculating multiple technical indicators, batch processing provides dramatic performance improvements:

66.7x More Efficient Than Individual Indicators

  • Individual indicators: GPU beneficial at 100K-1M rows per indicator
  • Batch processing: GPU beneficial at just 15K rows for 6+ indicators
  • Real-world applications almost always need multiple indicators
  • Single GPU transfer amortizes overhead across all calculations

The GPU Overhead Problem

Every GPU operation has fixed overhead costs:

  • Data transfer: CPU → GPU memory (~10-50ms)
  • Kernel launch: ~5-10μs per kernel
  • Memory allocation: GPU memory management
  • Data transfer back: GPU → CPU memory (~10-50ms)

For small datasets, this overhead exceeds the computation time!

How Batch Processing Solves This

Individual Indicators (6 separate calls):
┌─────────────────────────────────────────────────────────────┐
│ ATR:  Transfer → Compute → Transfer back                   │
│ RSI:  Transfer → Compute → Transfer back                   │
│ MACD: Transfer → Compute → Transfer back                   │
│ ...   (repeat for each indicator)                          │
│                                                             │
│ Total overhead: 6 × (2 transfers + kernel launches)       │
│ GPU beneficial at: 1M rows per indicator                   │
└─────────────────────────────────────────────────────────────┘

Batch Processing (1 call for all indicators):
┌─────────────────────────────────────────────────────────────┐
│ Transfer ALL data ONCE → Compute all 6 indicators → Return │
│                                                             │
│ Total overhead: 1 transfer pair + shared kernel launches  │
│ GPU beneficial at: 15K rows (66.7x lower threshold!)      │
└─────────────────────────────────────────────────────────────┘

Performance Numbers

Dataset Size Individual (Sequential) Batch Processing Speedup
10K rows ~15ms (CPU) ~3ms (CPU) 5x
50K rows ~75ms (CPU) ~12ms (GPU) 6.2x
100K rows ~150ms (CPU/GPU) ~20ms (GPU) 7.5x
500K rows ~750ms (GPU) ~85ms (GPU) 8.8x
1M rows ~1500ms (GPU) ~160ms (GPU) 9.4x

Batch processing is 2-3x faster than individual calls on CPU, and even better on GPU!


Basic Batch Usage

Import and Setup

import numpy as np
from kimsfinance.ops.batch import calculate_indicators_batch

# Your OHLCV data (numpy arrays, Polars Series, Pandas Series, or lists)
highs = np.array([102, 105, 104, 107, 106, 108, 107, 109])
lows = np.array([100, 101, 102, 104, 103, 105, 104, 106])
closes = np.array([101, 103, 102, 106, 104, 107, 105, 108])
volumes = np.array([1000, 1200, 900, 1500, 1100, 1300, 950, 1400])

Calculate All Indicators at Once

# One call calculates 6 indicators simultaneously
results = calculate_indicators_batch(
    highs=highs,
    lows=lows,
    closes=closes,
    volumes=volumes,
    engine="auto",      # Smart CPU/GPU selection
    streaming=None      # Auto-enable for large datasets
)

# Extract individual indicators from results dictionary
atr = results["atr"]                              # Average True Range
rsi = results["rsi"]                              # Relative Strength Index (0-100)
stoch_k, stoch_d = results["stochastic"]          # Stochastic Oscillator (%K, %D)
bb_upper, bb_mid, bb_lower = results["bollinger"] # Bollinger Bands
obv = results["obv"]                              # On Balance Volume
macd_line, signal, histogram = results["macd"]    # MACD components

Interpreting Results

All results are numpy arrays with the same length as your input data:

# Check shapes
print(f"Input data: {len(closes)} candles")
print(f"ATR shape: {atr.shape}")           # (8,) - same length
print(f"RSI shape: {rsi.shape}")           # (8,) - same length
print(f"Stochastic shapes: {stoch_k.shape}, {stoch_d.shape}")  # (8,), (8,)

# First few values will be NaN (warmup period)
print(f"ATR first 14 values: {atr[:14]}")  # [nan, nan, ... valid values]
print(f"RSI first 14 values: {rsi[:14]}")  # [nan, nan, ... valid values]

# Valid values start after warmup period
print(f"ATR value at index 20: {atr[20]:.4f}")   # Valid ATR
print(f"RSI value at index 20: {rsi[20]:.2f}")   # Valid RSI (0-100)

Supported Indicators

Batch processing calculates these 6 indicators with standard periods:

1. ATR (Average True Range)

  • Period: 14
  • Purpose: Measures volatility
  • Range: Positive values in price units
  • Formula: Wilder's smoothing of True Range
results = calculate_indicators_batch(highs, lows, closes)
atr = results["atr"]

# Typical usage
current_atr = atr[-1]  # Most recent ATR
avg_atr = np.nanmean(atr)  # Average ATR (skip NaN)
print(f"Current volatility: {current_atr:.2f}")

2. RSI (Relative Strength Index)

  • Period: 14
  • Purpose: Momentum oscillator (overbought/oversold)
  • Range: 0-100 (>70 = overbought, <30 = oversold)
  • Formula: 100 - (100 / (1 + RS)), RS = avg gain / avg loss
rsi = results["rsi"]

# Identify overbought/oversold conditions
overbought = rsi[-1] > 70
oversold = rsi[-1] < 30

print(f"Current RSI: {rsi[-1]:.2f}")
if overbought:
    print("Signal: Potential sell opportunity")
elif oversold:
    print("Signal: Potential buy opportunity")

3. Stochastic Oscillator

  • Period: 14, %D smoothing: 3
  • Purpose: Compare close to recent price range
  • Range: 0-100 (>80 = overbought, <20 = oversold)
  • Returns: (%K, %D) - %K is fast, %D is slow (smoothed)
stoch_k, stoch_d = results["stochastic"]

# Stochastic crossover signals
k_above_d = stoch_k[-1] > stoch_d[-1]  # Bullish when %K crosses above %D
k_below_d = stoch_k[-1] < stoch_d[-1]  # Bearish when %K crosses below %D

print(f"Stochastic: %K={stoch_k[-1]:.2f}, %D={stoch_d[-1]:.2f}")

4. Bollinger Bands

  • Period: 20, Std Dev: 2.0
  • Purpose: Volatility bands around price
  • Returns: (upper, middle, lower)
  • Formula: Middle = SMA(20), Upper/Lower = Middle ± 2×StdDev
bb_upper, bb_middle, bb_lower = results["bollinger"]

current_price = closes[-1]
bandwidth = bb_upper[-1] - bb_lower[-1]
percent_b = (current_price - bb_lower[-1]) / bandwidth

print(f"Bollinger Bands: Upper={bb_upper[-1]:.2f}, Lower={bb_lower[-1]:.2f}")
print(f"Price position: {percent_b:.2%} of bandwidth")

5. OBV (On Balance Volume)

  • Purpose: Cumulative volume based on price direction
  • Range: Unbounded (cumulative)
  • Formula: If price up: OBV += volume, If price down: OBV -= volume
# OBV requires volumes parameter
obv = results["obv"]  # None if volumes not provided

# OBV trend analysis
obv_sma = np.convolve(obv, np.ones(10)/10, mode='valid')  # 10-period SMA
obv_rising = obv[-1] > obv_sma[-1]  # OBV above its moving average

print(f"Current OBV: {obv[-1]:,.0f}")

6. MACD (Moving Average Convergence Divergence)

  • Periods: Fast=12, Slow=26, Signal=9
  • Purpose: Trend following momentum indicator
  • Returns: (macd_line, signal_line, histogram)
  • Formula: MACD = EMA(12) - EMA(26), Signal = EMA(MACD, 9)
macd_line, signal_line, histogram = results["macd"]

# MACD signals
bullish_cross = (macd_line[-2] < signal_line[-2] and
                 macd_line[-1] > signal_line[-1])
bearish_cross = (macd_line[-2] > signal_line[-2] and
                 macd_line[-1] < signal_line[-1])

print(f"MACD: {macd_line[-1]:.2f}, Signal: {signal_line[-1]:.2f}")
print(f"Histogram: {histogram[-1]:.2f}")

Complete Working Example

Backtesting System with Batch Processing

import numpy as np
import polars as pl
from kimsfinance.ops.batch import calculate_indicators_batch

# Load historical data (Polars is fastest)
df = pl.read_parquet("btc_1h_2023.parquet")

# Extract OHLCV arrays
highs = df["high"].to_numpy()
lows = df["low"].to_numpy()
closes = df["close"].to_numpy()
volumes = df["volume"].to_numpy()

print(f"Loaded {len(closes):,} candles")

# Calculate all indicators in one batch call
results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    engine="auto",      # GPU at 15K+ rows
    streaming=None      # Auto-enable at 500K+ rows
)

# Extract indicators
atr = results["atr"]
rsi = results["rsi"]
stoch_k, stoch_d = results["stochastic"]
bb_upper, bb_middle, bb_lower = results["bollinger"]
obv = results["obv"]
macd_line, signal_line, histogram = results["macd"]

# Simple trading strategy
signals = []
for i in range(100, len(closes)):  # Start after warmup
    # Multiple indicator confirmation
    rsi_oversold = rsi[i] < 30
    stoch_oversold = stoch_k[i] < 20
    macd_bullish = macd_line[i] > signal_line[i]
    price_below_bb = closes[i] < bb_lower[i]

    if rsi_oversold and stoch_oversold and macd_bullish:
        signals.append(("BUY", i, closes[i]))

    rsi_overbought = rsi[i] > 70
    stoch_overbought = stoch_k[i] > 80
    macd_bearish = macd_line[i] < signal_line[i]
    price_above_bb = closes[i] > bb_upper[i]

    if rsi_overbought and stoch_overbought and macd_bearish:
        signals.append(("SELL", i, closes[i]))

print(f"\nGenerated {len(signals)} trading signals")
for signal_type, index, price in signals[:5]:
    print(f"{signal_type} at index {index}: ${price:,.2f}")

Advanced Batch Patterns

1. Batch + Custom Indicators

Combine batch processing with your own custom indicators:

# Get batch indicators
results = calculate_indicators_batch(highs, lows, closes, volumes)

# Add custom indicator
def custom_momentum(closes, period=10):
    """Simple momentum: current price / price N periods ago."""
    momentum = np.zeros_like(closes)
    momentum[period:] = closes[period:] / closes[:-period]
    return momentum

# Combine batch results with custom indicators
momentum = custom_momentum(closes, period=10)

# Use all indicators together
signals = (
    (results["rsi"] < 30) &           # RSI oversold
    (results["stochastic"][0] < 20) & # Stochastic oversold
    (momentum > 1.0)                  # Positive momentum
)

print(f"Combined signals: {np.sum(signals)} opportunities")

2. Batch in Backtesting Systems

Efficient backtesting with batch indicators:

class Backtester:
    def __init__(self, initial_capital=10000):
        self.capital = initial_capital
        self.position = 0
        self.trades = []

    def run(self, highs, lows, closes, volumes):
        # Calculate all indicators once (fast!)
        indicators = calculate_indicators_batch(
            highs, lows, closes, volumes,
            engine="auto"
        )

        # Extract what we need
        rsi = indicators["rsi"]
        stoch_k, stoch_d = indicators["stochastic"]
        macd_line, signal_line, _ = indicators["macd"]

        # Iterate through bars
        for i in range(100, len(closes)):
            # Entry logic
            if self.position == 0:
                if (rsi[i] < 30 and stoch_k[i] < 20 and
                    macd_line[i] > signal_line[i]):
                    # Buy signal
                    self.position = self.capital / closes[i]
                    self.trades.append(("BUY", i, closes[i]))

            # Exit logic
            elif self.position > 0:
                if (rsi[i] > 70 or stoch_k[i] > 80 or
                    macd_line[i] < signal_line[i]):
                    # Sell signal
                    self.capital = self.position * closes[i]
                    self.trades.append(("SELL", i, closes[i]))
                    self.position = 0

        return self.capital, self.trades

# Run backtest
backtester = Backtester(initial_capital=10000)
final_capital, trades = backtester.run(highs, lows, closes, volumes)

print(f"Starting capital: $10,000")
print(f"Ending capital: ${final_capital:,.2f}")
print(f"Return: {(final_capital/10000 - 1)*100:.2f}%")
print(f"Number of trades: {len(trades)}")

3. Batch in Dashboards (Parallel Execution)

Calculate indicators for multiple symbols in parallel:

from multiprocessing import Pool
import polars as pl

def calculate_indicators_for_symbol(symbol):
    """Calculate indicators for a single symbol."""
    # Load data
    df = pl.read_parquet(f"data/{symbol}_1h.parquet")

    # Extract OHLCV
    highs = df["high"].to_numpy()
    lows = df["low"].to_numpy()
    closes = df["close"].to_numpy()
    volumes = df["volume"].to_numpy()

    # Batch calculate
    results = calculate_indicators_batch(
        highs, lows, closes, volumes,
        engine="auto"
    )

    return {
        "symbol": symbol,
        "current_price": closes[-1],
        "rsi": results["rsi"][-1],
        "atr": results["atr"][-1],
        "macd": results["macd"][0][-1],  # MACD line
        "obv": results["obv"][-1],
    }

# Calculate for multiple symbols in parallel
symbols = ["BTCUSDT", "ETHUSDT", "BNBUSDT", "SOLUSDT", "ADAUSDT"]

with Pool(processes=4) as pool:
    dashboard_data = pool.map(calculate_indicators_for_symbol, symbols)

# Display dashboard
for data in dashboard_data:
    print(f"\n{data['symbol']}: ${data['current_price']:,.2f}")
    print(f"  RSI: {data['rsi']:.2f}")
    print(f"  ATR: {data['atr']:.2f}")
    print(f"  MACD: {data['macd']:.2f}")
    print(f"  OBV: {data['obv']:,.0f}")

4. Streaming Large Datasets

Process massive datasets without running out of memory:

# 1 year of 1-minute data = 525,600 rows = ~40MB base data
# Without streaming: ~200MB working memory
# With streaming: Constant ~500MB chunks

# Load massive dataset
df = pl.read_parquet("btc_1m_2023.parquet")  # 525K rows

highs = df["high"].to_numpy()
lows = df["low"].to_numpy()
closes = df["close"].to_numpy()
volumes = df["volume"].to_numpy()

print(f"Processing {len(closes):,} candles (~40MB data)")

# Streaming auto-enables at 500K+ rows
results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    engine="auto",
    streaming=None  # Auto-enables at 500K+
)

print("✓ Processed without OOM")
print(f"Result sizes: ATR={results['atr'].nbytes/1024/1024:.1f}MB")

# For even larger datasets, force streaming
# 5 years of 1-minute data = 2.6M rows
results_huge = calculate_indicators_batch(
    huge_highs, huge_lows, huge_closes, huge_volumes,
    engine="auto",
    streaming=True  # Force streaming for safety
)

Performance Optimization

When to Use Batch vs Individual

Use Batch Processing When:

  • ✅ You need 2+ indicators (batch is always faster)
  • ✅ Dataset is 15K+ rows (GPU becomes beneficial)
  • ✅ Building dashboards (multiple symbols)
  • ✅ Running backtests (need multiple confirmations)
  • ✅ Real-time analysis (amortizes overhead)

Use Individual Indicators When:

  • ❌ You only need ONE indicator (rare!)
  • ❌ You need custom periods (batch uses fixed periods)
  • ❌ Dataset is <1K rows (overhead doesn't matter)

Comparison: Bad vs Good

# ❌ BAD: Sequential individual (slow, GPU at 1M rows)
import time
start = time.time()

atr = calculate_atr(highs, lows, closes, 14, engine='auto')
rsi = calculate_rsi(closes, 14, engine='auto')
stoch_k, stoch_d = calculate_stochastic_oscillator(
    highs, lows, closes, 14, engine='auto'
)
bb_upper, bb_mid, bb_lower = calculate_bollinger_bands(
    closes, 20, 2.0, engine='auto'
)
obv = calculate_obv(closes, volumes, engine='auto')
macd_line, signal, hist = calculate_macd(
    closes, 12, 26, 9, engine='auto'
)

elapsed_bad = time.time() - start
print(f"Sequential individual: {elapsed_bad*1000:.2f}ms")

# ✅ GOOD: Batch processing (66.7x efficient, GPU at 15K rows)
start = time.time()

results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    engine='auto'
)

elapsed_good = time.time() - start
print(f"Batch processing: {elapsed_good*1000:.2f}ms")
print(f"Speedup: {elapsed_bad/elapsed_good:.1f}x faster!")

Streaming Mode

Streaming automatically chunks data to prevent out-of-memory errors:

# Auto-enable at 500K rows
results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    streaming=None  # Auto-enables at 500K+ rows
)

# Force streaming for memory-constrained systems
results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    streaming=True  # Always use streaming
)

# Disable streaming for small datasets (faster)
results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    streaming=False  # Only for <500K rows
)

Memory Usage:

Dataset Size Without Streaming With Streaming
10K rows ~5MB ~5MB
100K rows ~50MB ~50MB
500K rows ~250MB ~500MB chunks
1M rows ~500MB ~500MB chunks
5M rows ~2.5GB (OOM!) ~500MB chunks

Memory Management

# Process multiple large datasets sequentially
symbols = ["BTC", "ETH", "BNB", "SOL", "ADA"]

for symbol in symbols:
    # Load data
    df = pl.read_parquet(f"data/{symbol}_1m.parquet")

    # Calculate indicators with streaming
    results = calculate_indicators_batch(
        df["high"].to_numpy(),
        df["low"].to_numpy(),
        df["close"].to_numpy(),
        df["volume"].to_numpy(),
        streaming=True  # Prevents OOM
    )

    # Process results
    save_results(symbol, results)

    # Explicitly free memory
    del df, results
    import gc
    gc.collect()

Parallel Batch Execution

Combine batch processing with parallel execution for maximum throughput:

from multiprocessing import Pool
import numpy as np

def process_window(args):
    """Process one window of data."""
    window_id, highs, lows, closes, volumes = args

    results = calculate_indicators_batch(
        highs, lows, closes, volumes,
        engine="auto",
        streaming=False  # Each window is small
    )

    return window_id, results

# Split data into windows
window_size = 5000
num_windows = len(closes) // window_size

windows = []
for i in range(num_windows):
    start = i * window_size
    end = start + window_size
    windows.append((
        i,
        highs[start:end],
        lows[start:end],
        closes[start:end],
        volumes[start:end]
    ))

# Process windows in parallel
with Pool(processes=8) as pool:
    results = pool.map(process_window, windows)

print(f"Processed {num_windows} windows in parallel")

Real-World Use Cases

1. Cryptocurrency Trading Dashboard

import polars as pl
from datetime import datetime

class CryptoDashboard:
    def __init__(self, symbols):
        self.symbols = symbols
        self.data = {}

    def update(self):
        """Refresh all indicators for all symbols."""
        for symbol in self.symbols:
            # Fetch latest data (last 1000 candles)
            df = self.fetch_ohlcv(symbol, limit=1000)

            # Batch calculate indicators
            results = calculate_indicators_batch(
                df["high"].to_numpy(),
                df["low"].to_numpy(),
                df["close"].to_numpy(),
                df["volume"].to_numpy(),
                engine="auto"  # GPU if available
            )

            # Store latest values
            self.data[symbol] = {
                "timestamp": datetime.now(),
                "price": df["close"][-1],
                "rsi": results["rsi"][-1],
                "atr": results["atr"][-1],
                "stoch_k": results["stochastic"][0][-1],
                "macd": results["macd"][0][-1],
                "signal": self.generate_signal(results)
            }

    def generate_signal(self, results):
        """Generate trading signal from indicators."""
        rsi = results["rsi"][-1]
        stoch_k = results["stochastic"][0][-1]
        macd_line = results["macd"][0][-1]
        signal_line = results["macd"][1][-1]

        if rsi < 30 and stoch_k < 20 and macd_line > signal_line:
            return "STRONG BUY"
        elif rsi < 40 and stoch_k < 30:
            return "BUY"
        elif rsi > 70 and stoch_k > 80 and macd_line < signal_line:
            return "STRONG SELL"
        elif rsi > 60 and stoch_k > 70:
            return "SELL"
        else:
            return "HOLD"

    def fetch_ohlcv(self, symbol, limit):
        # Implement your data fetching logic
        pass

# Usage
dashboard = CryptoDashboard(["BTCUSDT", "ETHUSDT", "SOLUSDT"])
dashboard.update()

for symbol, data in dashboard.data.items():
    print(f"{symbol}: {data['signal']} | RSI: {data['rsi']:.1f}")

2. Live Dashboard Updates

Real-time indicator updates with WebSocket data:

import asyncio
import websockets
from collections import deque

class LiveIndicators:
    def __init__(self, lookback=1000):
        self.lookback = lookback
        self.highs = deque(maxlen=lookback)
        self.lows = deque(maxlen=lookback)
        self.closes = deque(maxlen=lookback)
        self.volumes = deque(maxlen=lookback)

    def update(self, candle):
        """Add new candle and recalculate indicators."""
        self.highs.append(candle["high"])
        self.lows.append(candle["low"])
        self.closes.append(candle["close"])
        self.volumes.append(candle["volume"])

        if len(self.closes) >= 100:  # Minimum data
            # Recalculate all indicators
            results = calculate_indicators_batch(
                np.array(self.highs),
                np.array(self.lows),
                np.array(self.closes),
                np.array(self.volumes),
                engine="cpu",  # Fast enough for 1K candles
                streaming=False
            )

            return {
                "rsi": results["rsi"][-1],
                "atr": results["atr"][-1],
                "macd": results["macd"][0][-1],
                "signal": results["macd"][1][-1],
            }

        return None

# WebSocket handler
async def handle_kline_stream(uri):
    live = LiveIndicators(lookback=1000)

    async with websockets.connect(uri) as ws:
        while True:
            msg = await ws.recv()
            candle = parse_kline_message(msg)

            indicators = live.update(candle)
            if indicators:
                print(f"RSI: {indicators['rsi']:.1f} | "
                      f"MACD: {indicators['macd']:.2f}")

# Run
asyncio.run(handle_kline_stream("wss://stream.binance.com:9443/ws/btcusdt@kline_1m"))

3. Batch Report Generation

Generate indicator reports for multiple timeframes:

def generate_indicator_report(symbol, timeframes=["1h", "4h", "1d"]):
    """Generate multi-timeframe indicator report."""
    report = {"symbol": symbol, "timeframes": {}}

    for tf in timeframes:
        # Load data for timeframe
        df = load_ohlcv(symbol, timeframe=tf, limit=1000)

        # Calculate all indicators
        results = calculate_indicators_batch(
            df["high"].to_numpy(),
            df["low"].to_numpy(),
            df["close"].to_numpy(),
            df["volume"].to_numpy(),
            engine="auto"
        )

        # Extract key metrics
        report["timeframes"][tf] = {
            "current_price": df["close"][-1],
            "rsi": {
                "current": results["rsi"][-1],
                "signal": "Overbought" if results["rsi"][-1] > 70
                         else "Oversold" if results["rsi"][-1] < 30
                         else "Neutral"
            },
            "macd": {
                "line": results["macd"][0][-1],
                "signal": results["macd"][1][-1],
                "trend": "Bullish" if results["macd"][0][-1] > results["macd"][1][-1]
                         else "Bearish"
            },
            "bollinger": {
                "upper": results["bollinger"][0][-1],
                "lower": results["bollinger"][2][-1],
                "position": "Above" if df["close"][-1] > results["bollinger"][0][-1]
                           else "Below" if df["close"][-1] < results["bollinger"][2][-1]
                           else "Inside"
            }
        }

    return report

# Generate report
report = generate_indicator_report("BTCUSDT", timeframes=["1h", "4h", "1d"])

print(f"\nIndicator Report: {report['symbol']}")
for tf, data in report["timeframes"].items():
    print(f"\n{tf} Timeframe:")
    print(f"  Price: ${data['current_price']:,.2f}")
    print(f"  RSI: {data['rsi']['current']:.1f} ({data['rsi']['signal']})")
    print(f"  MACD: {data['macd']['trend']}")
    print(f"  Bollinger: {data['bollinger']['position']} bands")

4. Data Pipeline Integration

Integrate batch processing into your data pipeline:

import polars as pl
from pathlib import Path

class IndicatorPipeline:
    def __init__(self, input_dir, output_dir):
        self.input_dir = Path(input_dir)
        self.output_dir = Path(output_dir)
        self.output_dir.mkdir(exist_ok=True)

    def process_all_symbols(self):
        """Process all Parquet files in input directory."""
        parquet_files = list(self.input_dir.glob("*.parquet"))

        print(f"Processing {len(parquet_files)} symbols...")

        for file in parquet_files:
            symbol = file.stem
            self.process_symbol(symbol, file)

    def process_symbol(self, symbol, input_file):
        """Process single symbol and save enriched data."""
        # Load OHLCV data
        df = pl.read_parquet(input_file)

        # Calculate all indicators
        results = calculate_indicators_batch(
            df["high"].to_numpy(),
            df["low"].to_numpy(),
            df["close"].to_numpy(),
            df["volume"].to_numpy(),
            engine="auto",
            streaming=True  # Safe for large files
        )

        # Add indicators to DataFrame
        enriched_df = df.with_columns([
            pl.Series("atr", results["atr"]),
            pl.Series("rsi", results["rsi"]),
            pl.Series("stoch_k", results["stochastic"][0]),
            pl.Series("stoch_d", results["stochastic"][1]),
            pl.Series("bb_upper", results["bollinger"][0]),
            pl.Series("bb_middle", results["bollinger"][1]),
            pl.Series("bb_lower", results["bollinger"][2]),
            pl.Series("obv", results["obv"]),
            pl.Series("macd_line", results["macd"][0]),
            pl.Series("macd_signal", results["macd"][1]),
            pl.Series("macd_histogram", results["macd"][2]),
        ])

        # Save enriched data
        output_file = self.output_dir / f"{symbol}_enriched.parquet"
        enriched_df.write_parquet(output_file)

        print(f"✓ {symbol}: {len(df)} candles processed")

# Run pipeline
pipeline = IndicatorPipeline(
    input_dir="data/raw",
    output_dir="data/enriched"
)
pipeline.process_all_symbols()

Error Handling

Common Issues and Solutions

from kimsfinance.core.exceptions import (
    ConfigurationError,
    GPUNotAvailableError
)

try:
    results = calculate_indicators_batch(
        highs, lows, closes, volumes,
        engine="gpu"  # Force GPU
    )
except GPUNotAvailableError as e:
    # GPU not available, fallback to CPU
    print(f"GPU unavailable: {e}")
    results = calculate_indicators_batch(
        highs, lows, closes, volumes,
        engine="cpu"
    )

except ValueError as e:
    # Input validation error
    print(f"Invalid input: {e}")
    # Check: mismatched lengths, insufficient data, etc.

except ConfigurationError as e:
    # Invalid engine parameter
    print(f"Configuration error: {e}")

Input Validation

def validate_and_calculate(highs, lows, closes, volumes=None):
    """Safely calculate indicators with validation."""
    # Check minimum length
    if len(closes) < 100:
        raise ValueError(f"Need at least 100 candles, got {len(closes)}")

    # Check matching lengths
    if not (len(highs) == len(lows) == len(closes)):
        raise ValueError("OHLC arrays must have same length")

    if volumes is not None and len(volumes) != len(closes):
        raise ValueError("Volumes must match OHLC length")

    # Check for invalid values
    if np.any(np.isnan(closes[-50:])):  # Check recent data
        print("Warning: NaN values in recent data")

    # Calculate with appropriate settings
    results = calculate_indicators_batch(
        highs, lows, closes, volumes,
        engine="auto",
        streaming=len(closes) >= 500_000  # Auto-streaming
    )

    return results

Performance Benchmarks

Measured on Intel i9-13980HX (24 cores) + NVIDIA RTX 3500 Ada:

Dataset Size vs Performance

10K rows:
  Individual sequential: 15.2ms (CPU)
  Batch processing:      3.1ms (CPU)
  Speedup: 4.9x

50K rows:
  Individual sequential: 76.8ms (CPU/GPU)
  Batch processing:      12.4ms (GPU)
  Speedup: 6.2x

100K rows:
  Individual sequential: 153.5ms (GPU)
  Batch processing:      19.8ms (GPU)
  Speedup: 7.8x

500K rows:
  Individual sequential: 768.2ms (GPU)
  Batch processing:      87.3ms (GPU)
  Speedup: 8.8x

1M rows:
  Individual sequential: 1,534.6ms (GPU)
  Batch processing:      163.2ms (GPU)
  Speedup: 9.4x

Memory Efficiency

Dataset: 1M rows OHLCV (40MB base data)

Individual indicators (6 calls):
  Peak memory: ~360MB
  Data transfers: 6 × (40MB → GPU → 40MB back)
  Total transfer: 480MB

Batch processing (1 call):
  Peak memory: ~200MB
  Data transfers: 1 × (40MB → GPU → 40MB back)
  Total transfer: 80MB
  Memory saved: 44%

Summary

Key Takeaways:

  1. Always use batch processing when you need multiple indicators
  2. 66.7x more efficient - GPU beneficial at 15K vs 1M rows
  3. Use engine="auto" for smart CPU/GPU selection
  4. Enable streaming for datasets >500K rows (prevents OOM)
  5. Batch is 2-3x faster even on CPU-only systems

Quick Reference:

# Import
from kimsfinance.ops.batch import calculate_indicators_batch

# Calculate
results = calculate_indicators_batch(
    highs, lows, closes, volumes,
    engine="auto",      # Smart selection
    streaming=None      # Auto-enable at 500K+
)

# Extract
atr = results["atr"]
rsi = results["rsi"]
stoch_k, stoch_d = results["stochastic"]
bb_upper, bb_mid, bb_lower = results["bollinger"]
obv = results["obv"]
macd_line, signal, histogram = results["macd"]

Next Steps:


Built with ⚡ for blazing-fast technical analysis.