finlab.ml

Machine learning module providing feature engineering, label generation, and model training integration.

Use Cases

Build feature sets for ML stock selection strategies
Generate technical indicators (TA-Lib integration)
Design training labels (returns, risk metrics)
Integrate with the qlib framework for model training
Predict future stock performance

Quick Examples

Feature Engineering

from finlab import data
from finlab.ml import feature as mlf

data.set_market('us')
with data.us_universe(index='SP500'):
    raw_close = data.get('price:close')  # unadjusted price for valuation
    market_value = data.get('etl:us_market_value')
    eps = data.get('us_income_statement:eps_diluted')
    net_income = data.get('us_income_statement:net_income')
    equity = data.get('us_balance_sheet:total_stockholders_equity')

    # Combine fundamental features (ratios only where the denominator is positive)
    ttm_eps = eps.rolling(4).sum()
    features = mlf.combine(
        {
            'pb': market_value / equity[equity > 0],
            'pe': raw_close / ttm_eps[ttm_eps > 0],  # Trailing-12-month P/E
            'roe': net_income / equity[equity > 0],
        },
        resample='W',
    )

    # Add technical indicators
    indicators = [
        'talib.RSI__real__timeperiod__14__',
        'talib.MACD__macd__fastperiod__12__slowperiod__26__signalperiod__9__',
    ]
    features_ta = mlf.combine(
        {'fundamental': features, 'technical': mlf.ta(indicators)}, resample='W'
    )

Label Generation

from finlab.ml import label as mll

# Generate next-week return labels
label = mll.return_percentage(features.index, resample='W', period=1)

qlib Model Training

import finlab.ml.qlib as q

# Split train/test sets, keeping rows where the features and label are known
complete = features.notna().all(axis=1) & label.notna()
is_train = features.index.get_level_values('datetime') < '2023-01-01'
X_train, y_train = features[complete & is_train], label[complete & is_train]
X_test = features[~is_train].dropna()

# Train LightGBM model
model = q.LGBModel()
model.fit(X_train, y_train)

# Predict and convert to position weights
pred = model.predict(X_test)
position = pred.is_largest(30)  # Buy top 30

Detailed Guide

See Machine Learning Strategy Development for: - Complete ML strategy development workflow - Feature engineering best practices - Label design techniques - Model training and optimization - Overfitting prevention

API Reference

finlab.ml.feature

Feature engineering module for combining and processing various features.

combine()

combine(features, resample=None, sample_filter=None, **kwargs) -> pd.DataFrame

Combines a dictionary of features into one pandas DataFrame indexed by (datetime, instrument), with one column per feature. Wide features (dates by symbols) are reindexed onto the union of their dates and the symbols they share, forward-filled, then stacked; features that already have a (datetime, instrument) MultiIndex are concatenated as they are. Quarterly FinLab data is converted to filing dates first.

Parameter	Type	Description
`features`	`dict[str, pd.DataFrame or Callable]`	Feature name to a DataFrame (dates by symbols, or a MultiIndex frame), or a function returning one.
`resample`	`str`, `pd.DatetimeIndex` or `None`	Optional resampling. String frequencies use calendar period labels (e.g. Sunday for `'W'`); pass a `DatetimeIndex` for explicit trading-date anchors. MultiIndex features use the same resampling. Default `None` keeps the original dates.
`sample_filter`	`pd.DataFrame` or `None`	Boolean DataFrame (dates by symbols) selecting which samples to keep.
`**kwargs`		Passed to the resampler.

Returns: pd.DataFrame with all input features combined.

To anchor features on each week's last trading day and build labels on the same dates:

import pandas as pd
from finlab import data
from finlab.ml import feature as mlf, label as mll

data.set_market('us')
rsi = data.indicator('RSI')
calendar = rsi.index
weekly = pd.DatetimeIndex(calendar.to_series().resample('W').last().dropna())
features = mlf.combine({'rsi': rsi}, resample=weekly)
labels = mll.return_percentage(features.index, resample=weekly, period=2)

Usage Examples:

from finlab import data
from finlab.ml import feature as mlf

data.set_market('us')
with data.us_universe(index='SP500'):
    close = data.get('price:adj_close')
    raw_close = data.get('price:close')  # unadjusted price for valuation
    market_value = data.get('etl:us_market_value')
    eps = data.get('us_income_statement:eps_diluted')
    net_income = data.get('us_income_statement:net_income')
    equity = data.get('us_balance_sheet:total_stockholders_equity')

ttm_eps = eps.rolling(4).sum()
pb = market_value / equity[equity > 0]
pe = raw_close / ttm_eps[ttm_eps > 0]  # Trailing-12-month P/E

# Example 1: Combine fundamental features
features = mlf.combine(
    {
        'pb': pb,
        'pe': pe,
        'roe': net_income / equity[equity > 0],
    },
    resample='W',
)

# Example 2: Combine technical indicators
indicators = [
    'talib.RSI__real__timeperiod__14__',
    'talib.MACD__macd__fastperiod__12__slowperiod__26__signalperiod__9__',
]
with data.us_universe(index='SP500'):
    features = mlf.combine({'talib': mlf.ta(indicators)}, resample='D')

# Example 3: Mix multiple feature types
with data.us_universe(index='SP500'):
    technical = mlf.ta(indicators)
features = mlf.combine(
    {
        'fundamental': mlf.combine({'pb': pb, 'pe': pe}),
        'technical': technical,
        'custom': close / close.average(20),  # Price relative to its 20-day average
    },
    resample='W',
)

resample Parameter

'D' - Daily
'W' - Weekly (labeled on the calendar week end, Sunday)
'M' - Monthly (month-end)
Features and labels must use the same resample frequency!

ta()

finlab.ml.feature.ta

ta(
    feature_names=None,
    factories=None,
    resample=None,
    start_time=None,
    end_time=None,
    adj=False,
    cpu=-1,
    **kwargs,
)

Calculate technical indicator values for a list of feature names.

PARAMETER	DESCRIPTION
`feature_names`	A list of technical indicator feature names. Defaults to None. TYPE: `list[str] \| None` DEFAULT: `None`
`factories`	A dictionary of factories to generate technical indicators. Defaults to {"talib": TalibIndicatorFactory()}. TYPE: `dict[str, TalibIndicatorFactory] \| None` DEFAULT: `None`
`resample`	The frequency to resample the data to. Defaults to None. TYPE: `str \| None` DEFAULT: `None`
`start_time`	The start time of the data. Defaults to None. TYPE: `str \| None` DEFAULT: `None`
`end_time`	The end time of the data. Defaults to None. TYPE: `str \| None` DEFAULT: `None`
`**kwargs`	Additional keyword arguments to pass to the resampler function. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`DataFrame`	pd.DataFrame: technical indicator feature names and their corresponding values.

Technical Indicator Computation:

from finlab.ml import feature as mlf

data.set_market('us')
with data.us_universe(index='SP500'):
    # Compute specific indicators
    indicators = mlf.ta(
        [
            'talib.RSI__real__timeperiod__14__',
            'talib.MACD__macd__fastperiod__12__slowperiod__26__signalperiod__9__',
            'talib.BBANDS__upperband__timeperiod__20__nbdevup__2.0__nbdevdn__2.0__matype__0__',
        ],
        resample='W',
    )

    # Random parameter sets for chosen indicators
    names = [
        name
        for name in mlf.ta_names(n=3)
        if name.startswith(('talib.RSI__', 'talib.MACD__', 'talib.ATR__'))
    ]
    random_indicators = mlf.ta(names, resample='W')

ta_names()

finlab.ml.feature.ta_names

ta_names(lb=1, ub=10, n=1, factory=None)

Generate a list of technical indicator feature names.

PARAMETER	DESCRIPTION
`lb`	The lower bound of the multiplier of the default parameter for the technical indicators. TYPE: `int` DEFAULT: `1`
`ub`	The upper bound of the multiplier of the default parameter for the technical indicators. TYPE: `int` DEFAULT: `10`
`n`	The number of random samples for each technical indicator. TYPE: `int` DEFAULT: `1`
`factory`	A factory object to generate technical indicators. Defaults to TalibIndicatorFactory. TYPE: `IndicatorFactory` DEFAULT: `None`

RETURNS	DESCRIPTION
`list[str]`	list[str]: A list of technical indicator feature names.

Examples:

import finlab.ml.feature as f


# method 1: generate each indicator with random parameters
features = f.ta()

# method 2: generate specific indicator
feature_names = ['talib.MACD__macdhist__fastperiod__52__slowperiod__212__signalperiod__75__']
features = f.ta(feature_names, resample='W')

# method 3: generate some indicator
feature_names = f.ta_names()
features = f.ta(feature_names)

Generate Indicator Name List:

from finlab.ml import feature as mlf

# Generate all TA-Lib indicators with 10 random parameter sets each
indicator_names = mlf.ta_names(n=10)
print(f"Total {len(indicator_names)} indicators")

# View examples
for name in indicator_names[:5]:
    print(name)
# Output (parameters are drawn at random; indicators without parameters appear once):
# talib.HT_DCPERIOD__real__
# talib.HT_DCPHASE__real__
# talib.HT_PHASOR__inphase__
# talib.HT_PHASOR__quadrature__
# talib.HT_SINE__sine__

Recommended n Values

n=1: one random parameter set per indicator (about 130 names with TA-Lib 0.8)
n=5: up to 5 random parameter sets per indicator (about 560 names)
n=10: more variety but longer computation time (about 1,070 names)
Start with n=1 to test feasibility, then increase

Notes

Too many indicators can lead to:
- Long computation time
- High memory usage
- Increased overfitting risk
Recommend using feature selection to reduce indicator count

finlab.ml.label

Label generation module for designing machine learning target variables.

daytrading_percentage()

finlab.ml.label.daytrading_percentage

daytrading_percentage(index, resample='D', **kwargs)

Calculate the percentage change of market prices over a given period.

PARAMETER	DESCRIPTION
`index`	A multi-level index of datetime and instrument. TYPE: `Index`
`resample`	The resample frequency for the output data. Defaults to "D". TYPE: `str \| None` DEFAULT: `'D'`
`**kwargs`	Additional arguments to be passed to the resampler function. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Series`	pd.Series: A pd.Series containing the percentage change of stock prices.

Next-Day Open-to-Close Return (day-trading label):

from finlab.ml import feature as mlf, label as mll

# Build daily features
features = mlf.combine({...}, resample='D')

# Label: return from the next trading day's open to its close
label = mll.daytrading_percentage(features.index)

For forward returns over N periods, use mll.return_percentage():

# Generate label: predict next 1-week return
label = mll.return_percentage(features.index, resample='W', period=1)

# Generate label: predict next 4-week return
label_4w = mll.return_percentage(features.index, resample='W', period=4)

Choosing the period Parameter of return_percentage()

period=1: Short-term prediction (suitable for weekly/monthly rebalancing)
period=4: Medium-term prediction (suitable for quarterly rebalancing)
Larger period: More stable prediction but slower signal
Recommendation: Match with strategy rebalancing frequency

maximum_adverse_excursion()

finlab.ml.label.maximum_adverse_excursion

maximum_adverse_excursion(
    index, period=1, trade_at_price='close'
)

Calculate the maximum adverse excursion of market prices over a given period.

PARAMETER	DESCRIPTION
`index`	A multi-level index of datetime and instrument. TYPE: `Index`
`period`	The number of periods to calculate the percentage change over. Defaults to 1. TYPE: `int` DEFAULT: `1`
`trade_at_price`	The price for execution. Defaults to `close`. TYPE: `str` DEFAULT: `'close'`

RETURNS	DESCRIPTION
`Series`	pd.Series: A pd.Series containing the maximum adverse excursion.

Maximum Adverse Excursion (Risk Metric):

from finlab.ml import label as mll

# Compute maximum drawdown over next N periods
mae_label = mll.maximum_adverse_excursion(features.index, period=5)

# Can be used to train risk prediction models
# More negative values indicate higher risk

finlab.ml.qlib

qlib framework integration module providing multiple machine learning models for stock prediction.

Model Classes

finlab.ml.qlib provides the following models:

finlab.ml.qlib.LGBModel

LGBModel()

LGBModel is a wrapper model for LightGBM model.

import finlab.ml.qlib as q

# build X_train, y_train, X_test

model = q.LGBModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

finlab.ml.qlib.XGBModel

XGBModel()

XGBModel is a wrapper model for XGBoost model.

import finlab.ml.qlib as q

# build X_train, y_train, X_test

model = q.XGBModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

finlab.ml.qlib.CatBoostModel

CatBoostModel()

CatBoostModel is a wrapper model for CatBoost model.

import finlab.ml.qlib as q

# build X_train, y_train, X_test

model = q.CatBoostModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

finlab.ml.qlib.LinearModel

LinearModel()

LinearModel is a wrapper model for Linear model.

import finlab.ml.qlib as q

# build X_train, y_train, X_test

model = q.LinearModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Basic Usage Example:

from finlab import data
from finlab.ml import feature as mlf, label as mll
import finlab.ml.qlib as q

# 1. Prepare features
data.set_market('us')
with data.us_universe(index='SP500'):
    raw_close = data.get('price:close')  # unadjusted price for valuation
    market_value = data.get('etl:us_market_value')
    eps = data.get('us_income_statement:eps_diluted')
    equity = data.get('us_balance_sheet:total_stockholders_equity')

ttm_eps = eps.rolling(4).sum()
features = mlf.combine(
    {
        'pb': market_value / equity[equity > 0],
        'pe': raw_close / ttm_eps[ttm_eps > 0],
    },
    resample='W',
)

# 2. Prepare labels
label = mll.return_percentage(features.index, resample='W', period=1)

# 3. Split train/test sets, keeping rows where the features and label are known
complete = features.notna().all(axis=1) & label.notna()
is_train = features.index.get_level_values('datetime') < '2020-01-01'
X_train, y_train = features[complete & is_train], label[complete & is_train]
X_test = features[~is_train].dropna()

# 4. Build and train model
model = q.XGBModel()  # Options: q.LGBModel(), q.CatBoostModel(), q.LinearModel()
model.fit(X_train, y_train)

# 5. Predict
y_pred = model.predict(X_test)

Supported Model Types:

Model	Description	Pros	Cons
`LGBModel()`	LightGBM	Fast, good performance, low memory usage	Requires lightgbm
`XGBModel()`	XGBoost	Stable, high interpretability	Slower training
`CatBoostModel()`	CatBoost	Handles categorical features well, low overfitting risk	High memory usage
`LinearModel()`	Linear Regression	Simple, fast, low overfitting risk	Usually lower performance

Model Selection Recommendations

Beginners: LGBModel() (balanced speed and performance)
Stability-focused: XGBModel()
Categorical features: CatBoostModel()
Quick idea validation: LinearModel()
Avoid overfitting: Start with LinearModel() as baseline, then try complex models

Common Mistakes

Data leakage: Ensure train and test sets do not overlap in time (use temporal split, not random split)
Overfitting: Test set performance far worse than training set (IC < 0.02 may indicate overfitting)
Look-ahead bias: Labels should use .shift(-1) or period=1 to avoid leaking future information

Complete Example

Building an ML Stock Selection Strategy

from finlab import data
from finlab.ml import feature as mlf, label as mll
import finlab.ml.qlib as q
from finlab.backtest import sim

data.set_market('us')
with data.us_universe(index='SP500'):
    raw_close = data.get('price:close')  # unadjusted price for valuation
    market_value = data.get('etl:us_market_value')
    eps = data.get('us_income_statement:eps_diluted')
    net_income = data.get('us_income_statement:net_income')
    equity = data.get('us_balance_sheet:total_stockholders_equity')
    # Technical indicators (use few to avoid overfitting)
    technical = mlf.ta(mlf.ta_names(n=1)[:20])  # Only the first 20

# Step 1: Build feature set
print("Building features...")
ttm_eps = eps.rolling(4).sum()
features = mlf.combine(
    {
        # Fundamental features, only where the denominator is positive
        'pb': market_value / equity[equity > 0],
        'pe': raw_close / ttm_eps[ttm_eps > 0],  # Trailing-12-month P/E
        'roe': net_income / equity[equity > 0],
        'technical': technical,
    },
    resample='W',
)

# Step 2: Generate labels
print("Generating labels...")
label = mll.return_percentage(features.index, resample='W', period=1)

# Step 3: Split train/test sets
print("Splitting data...")
is_train = features.index.get_level_values('datetime') < '2023-01-01'
X_train, y_train = features[is_train], label[is_train]
X_test = features[~is_train]  # LightGBM handles missing values itself

# Step 4: Train model
print("Training model...")
model = q.LGBModel()
model.fit(X_train, y_train)

# Step 5: Predict
print("Generating predictions...")
pred = model.predict(X_test)

# Step 6: Convert to trading signals
print("Generating trading signals...")
position = pred.is_largest(30)  # Buy stocks with highest predicted returns

# Step 7: Backtest
print("Running backtest...")
report = sim(position, resample='W')
report.display()

FAQ

Q: What happens if feature and label resample frequencies don't match?

Mismatched frequencies cause date misalignment, preventing correct feature-label pairing:

# Wrong: Inconsistent resample
features = mlf.combine({...}, resample='W')  # Weekly
label = mll.return_percentage(features.index, resample='D', period=1)  # Daily
# -> Labels measure 1-day returns, not 1-week returns

# Correct: Consistent resample
features = mlf.combine({...}, resample='W')
label = mll.return_percentage(features.index, resample='W', period=1)

Q: How do I prevent overfitting?

# Method 1: Reduce feature count
features = mlf.ta(mlf.ta_names(n=1)[:20])  # Only 20 indicators

# Method 2: Use regularization (reduce model complexity)
# LightGBM supports L1/L2 regularization, tree depth limits, etc.
# Method 3: Use a simple model
model = q.LinearModel()  # Linear model is less prone to overfitting

# Method 4: Out-of-sample testing
# Ensure test set performance is close to training set (no more than 50% gap)

Q: Training is too slow, what can I do?

# Method 1: Reduce data range
data.truncate_start = '2020-01-01'

# Method 2: Reduce feature count
features = mlf.ta(mlf.ta_names(n=1)[:10])  # Only 10 indicators

# Method 3: Use resample='W' or 'M' (instead of 'D')
features = mlf.combine({...}, resample='W')  # Weekly data is faster

Q: How do I handle missing values?

# First check the missing value situation
print(f"Feature missing ratio: {features.isna().sum().sum() / features.size:.2%}")
print(f"Label missing ratio: {label.isna().sum() / len(label):.2%}")

# Method 1: Drop samples with missing values
features_clean = features.dropna()
label_clean = label.dropna()

# Method 2: Forward fill each stock's own history (suitable for time series)
features_filled = features.groupby(level='instrument').ffill()
label_filled = label.groupby(level='instrument').ffill()

# Method 3: Fill with 0 (suitable for technical indicators)
features_zero = features.fillna(0)

Q: How do I convert prediction results to trading signals?

# Method 1: Buy top N
position = pred.is_largest(30)

# Method 2: Set threshold
position = pred > pred.quantile(0.8)  # Buy top 20%

# Method 3: Long-short strategy
long_position = pred.is_largest(20)  # Long top 20
short_position = pred.is_smallest(20)  # Short bottom 20
position = long_position.astype(float) - short_position.astype(float)

# Method 4: Allocate weights by prediction rank (weights sum to 1 each date)
scores = pred.rank(axis=1, pct=True)
position = scores.div(scores.sum(axis=1), axis=0)

Resources

Machine Learning Strategy Workflow - End-to-end development guide
Machine Learning Strategy Development - Detailed tutorial
qlib Official Documentation - qlib framework docs
TA-Lib Indicator List - All technical indicators