Machine Learning

Library Installation

Required Installation

Install core packages (finlab, TA-Lib):

pip install finlab
pip install ta-lib

Model Libraries (Optional)

finlab.ml.qlib supports various models (LightGBM, XGBoost, CatBoost, PyTorch, TensorFlow, etc.). Install them as needed following their official documentation; most are pre-installed in Colab.

ML workflows are complex -- consider using an AI assistant

From feature engineering to model training, ML strategies involve many steps. After installing FinLab Skill, the AI coding assistant can help you select features, split datasets, train models, and interpret backtest results.

Feature Processing

Using the Combine Function to Merge Features

finlab.ml.feature.combine merges features from multiple sources (technical, fundamental, custom) with resampling support. Examples:

Merge P/B ratio and P/E ratio into one feature set (S&P 500 stocks, computed from SEC filings):

from finlab import data
from finlab.ml import feature as mlf

data.set_market('us')
with data.us_universe(index='SP500'):
    raw_close = data.get('price:close')  # unadjusted price for valuation
    market_value = data.get('etl:us_market_value')
    eps = data.get('us_income_statement:eps_diluted')
    equity = data.get('us_balance_sheet:total_stockholders_equity')

ttm_eps = eps.rolling(4).sum()  # Trailing-12-month EPS
features = mlf.combine(
    {
        # Ratios only where the denominator is positive
        'pb': market_value / equity[equity > 0],
        'pe': raw_close / ttm_eps[ttm_eps > 0],
    },
    resample='W',
)

features.head()

The result is indexed by (datetime, instrument), for example (2024-01-07, 'AAPL'), with one column per feature: pb and pe.

Merge technical indicators into one feature set:

from finlab import data
from finlab.ml import feature as mlf

data.set_market('us')
# Three random parameter sets each for RSI, MACD and ATR
names = [
    name
    for name in mlf.ta_names(n=3)
    if name.startswith(('talib.RSI__', 'talib.MACD__', 'talib.ATR__'))
]
with data.us_universe(index='SP500'):
    technical = mlf.combine({'talib': mlf.ta(names)}, resample='W')

The result has the same (datetime, instrument) index with one column per indicator output, named like talib.RSI__real__timeperiod__37__ and talib.MACD__macdsignal__fastperiod__48__slowperiod__104__signalperiod__36__. Indicators are computed from the prices of the active market and universe.

Using TA-Lib to Generate Technical Indicators

finlab exposes ta and ta_names for generating TA-Lib indicator features.

ta_names Function

ta_names generates a list of TA-Lib indicator names, one per parameter configuration.

n: random parameter configurations per indicator. n=10 yields up to 10 variants per TA-Lib indicator; indicators without parameters appear once.

from finlab.ml import feature as mlf

mlf.ta_names(n=1)

['talib.HT_DCPERIOD__real__',
 'talib.HT_DCPHASE__real__',
 'talib.HT_PHASOR__quadrature__',
 'talib.HT_PHASOR__inphase__',
 'talib.HT_SINE__sine__',
 'talib.HT_SINE__leadsine__'
 ...
 ]

ta Function

ta computes the actual values for a list of names produced by ta_names.

resample: optional parameter that resamples the computed indicator values to a given time frequency.

from finlab.ml import feature as mlf

mlf.ta(
    [
        'talib.HT_DCPERIOD__real__',
        'talib.HT_DCPHASE__real__',
        'talib.HT_PHASOR__quadrature__',
    ],
    resample='W',
)

With resample='W' the dates move to the weekly period labels (Sundays), one row per symbol per week.

Label Generation

Using the Label Function to Generate Labels

finlab.ml.label provides various return/risk label computations for training prediction models.

Predicting daytrading_percentage

This function computes the return from the next trading day's open to its close (a day-trading target).

resample: Must match the resample used in combine to align time periods.

from finlab.ml import label as mll

# Labels for the weekly features built above; use the same resample
label = mll.daytrading_percentage(features.index, resample='W')

The label is a pd.Series with the same (datetime, instrument) index as the features, so it can be paired with them row by row.

Predicting N-day Future Returns

Computes the percentage change within a given period for analyzing medium to long-term performance.

label = mll.return_percentage(features.index, resample='W', period=1)

Maximum Adverse Excursion

MAE: Maximum adverse movement during the holding period (currently does not support resample).

label = mll.maximum_adverse_excursion(features.index, period=1)

Maximum Favorable Excursion

MFE: Maximum favorable movement during the holding period (currently does not support resample).

label = mll.maximum_favorable_excursion(features.index, period=1)

Excess Over Median

Excess return relative to the market-wide median return for the same period.

label = mll.excess_over_median(features.index, resample='W', period=1)

Excess Over Mean

Excess return relative to the market-wide mean return for the same period.

label = mll.excess_over_mean(features.index, resample='W', period=1)

Ensure the index and market settings are correct; labels can be directly combined with features for model training.

Model Training with Qlib

WrapperModel adapts LightGBM, XGBoost, CatBoost, linear models, TabNet, DNN, etc. into a uniform fit/predict interface.

LGBModel

Wraps the LightGBM model.

import finlab.ml.qlib as q

# Next-week returns as labels; train on data before 2023, predict afterwards
labels = mll.return_percentage(features.index, resample='W', period=1)
complete = features.notna().all(axis=1) & labels.notna()
is_train = features.index.get_level_values('datetime') < '2023-01-01'
X_train = features[complete & is_train]
y_train = labels[complete & is_train]
X_test = features[~is_train].dropna()

model = q.LGBModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

XGBModel

Wraps the XGBoost model.

model = q.XGBModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

DEnsmbleModel

Wraps the Double Ensemble model.

model = q.DEnsmbleModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

CatBoostModel

Wraps the CatBoost model.

model = q.CatBoostModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

LinearModel

Wraps a linear model.

model = q.LinearModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

TabnetModel

Wraps the TabNet model.

model = q.TabnetModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

DNNModel

Wraps a deep neural network model.

model = q.DNNModel()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

The wrappers let you focus on features and strategy without getting bogged down in model details.

get_models

get_models quickly retrieves the list of available models and initializes them, facilitating multi-model experimentation.

import finlab.ml.qlib as q

# Get all available models
models = q.get_models()

# Print all model names
print(list(models.keys()))

# Select and instantiate a model, e.g., LightGBM
model = models['LGBModel']()

# Assuming X_train, y_train, X_test are already prepared

# Train model
model.fit(X_train, y_train)

# Predict using the trained model
y_pred = model.predict(X_test)

Running Backtests

Use sim to backtest, computing strategy returns and risk metrics:

from finlab.backtest import sim

position = y_pred.is_largest(50)

sim(position, resample='4W')