FinLab Data Flow In-Depth

Data Architecture Overview

graph TB
    A[Data Sources] --> B[GCS Bucket]
    B --> C[BigQuery]
    C --> D[finlab.data.get]
    D --> E{Local cache?}
    E -->|Yes| F[Return cached data]
    E -->|No| G[Download from BigQuery]
    G --> H[Store to cache]
    H --> F
    F --> I[FinlabDataFrame]
    I --> J[Strategy Development]

    style A fill:#e1f5ff
    style D fill:#fff4e1
    style F fill:#e8f5e9

Data Sources

US Stock Data

Category	Data Source	Update Frequency	Data Range
Price (`us_price`)	NYSE, NASDAQ and NYSE American listings	Daily, after the U.S. close	2016 to present
ETF and fund prices (`us_fund_price`)	U.S.-listed ETFs and mutual funds	Daily, after the U.S. close	2016 to present
Financial Statements (`us_income_statement`, `us_balance_sheet`, `us_cash_flow`)	SEC 10-Q / 10-K filings	After each filing; aligned to the filing date	2015 to present
Technical Indicators	Calculated by FinLab	Computed on request	Depends on price data

Taiwan Stock Data

Category	Data Source	Update Frequency	Data Range
Price	TWSE/TPEx	Daily at 18:00 Taiwan time	2007 to present
Financial Statements	MOPS (Market Observation Post System)	After each quarterly report	2000 to present
Monthly Revenue	MOPS	After the 10th of each month	2000 to present
Institutional Trading	TWSE	Daily at 18:00 Taiwan time	2010 to present

Data Update Schedule

U.S. prices and fundamentals update once a day after the U.S. market close. Taiwan data follows this timeline (Taiwan time):

gantt
    title Taiwan Stock Data Daily Update Timeline
    dateFormat HH:mm
    axisFormat %H:%M

    section Price Data
    Market close    :done, 13:30, 1m
    Scraper starts  :done, 18:00, 10m
    Data uploaded   :done, 18:10, 20m
    Available       :crit, 18:30, 1m

    section Chip Data
    Scraper starts  :done, 18:00, 30m
    Data uploaded   :done, 18:30, 30m
    Available       :crit, 19:00, 1m

    section Financial Statements
    Disclosure      :done, 08:00, 1m
    Scraper starts  :done, 09:00, 60m
    Data uploaded   :done, 10:00, 30m
    Available       :crit, 10:30, 1m

Data Caching Mechanism

Local Cache Directory

Downloaded datasets are cached in ~/finlab_db (in Google Colab, My Drive/Colab_tmp/finlab_db when Google Drive is mounted). To use another directory:

from finlab import data

data.set_storage(data.FileStorage('/path/to/custom/cache'))

Cache Update Logic

flowchart TD
    A[Call data.get] --> B{Local cache exists?}
    B -->|No| C[Full download from the cloud]
    B -->|Yes| D{Cache at the published revision?}
    D -->|Yes| E[Use cached data]
    D -->|No| H{Cache older than full_refresh_before?}
    H -->|Yes| C
    H -->|No| I[Merge .recent; full download on a hash mismatch]
    C --> F[Store locally, stamped with the published revision]
    I --> F
    F --> E
    E --> G[Return DataFrame]

Published revisions come from a shared index fdata publishes; each process checks it at most every five minutes and asks the signing service when the index is unavailable. Updated or corrected data arrives within about five minutes; just call data.get again (finlab 2.2.5+).

Free accounts receive different data from paid accounts (Taiwan history only up to the free cutoff), so their cache lives in a free subfolder of the default data directory, shared by every process on the machine and never read by a paid account. Free accounts skip the revision index: each read reports the cache's time to the signing service, and when no newer file exists the local cache is used without a download or any daily download quota; otherwise it downloads again.

Large full tables (such as broker_transactions and important_info_announcement) also publish a <dataset>.delta: a cache no older than its full_refresh_before downloads only the days that changed recently, replaces its rows on those days and verifies the whole-table hash; an older cache downloads the full table and then applies the delta. A hash mismatch falls back to a full download, so inconsistent data is never returned.

Cache Management

from finlab import data

# Release cached data from RAM (files on disk are kept)
data.free_memory()

# Delete the local cache and restore the default settings
data.clear()

Data Query Best Practices

1. Batch Load Data

Bad practice: Calling data.get() repeatedly

data.set_market('us')

# Loading the data again for every backtest (slow!)
for param in [20, 60, 120]:
    with data.us_universe(index='SP500'):
        close = data.get('price:adj_close')  # Redundant loading
    position = close > close.average(param)
    report = sim(position, resample='M')

Good practice: Load once, reuse multiple times

# Load only once
with data.us_universe(index='SP500'):
    close = data.get('price:adj_close')

# Reuse it for every backtest
for param in [20, 60, 120]:
    position = close > close.average(param)
    report = sim(position, resample='M')

2. Use Date Filtering

# Load only data after 2020
close = data.get('price:adj_close')
close_recent = close[close.index >= '2020-01-01']

# Or use loc
close_recent = close.loc['2020-01-01':]

3. Use Stock Filtering

# Load only specific stocks
close = data.get('price:adj_close')
close_subset = close[['AAPL', 'MSFT', 'NVDA']]

# Or use filter conditions
market_cap = data.get('etl:us_market_value')
large_cap = market_cap > 100_000_000_000  # Market cap > $100 billion
close_large = close[large_cap]

4. Avoid Redundant Computation

Bad practice

for stock in ['AAPL', 'MSFT', 'NVDA']:
    close = data.get('price:adj_close')[stock]
    ma20 = close.rolling(20).mean()  # Recomputed each time

Good practice

close = data.get('price:adj_close')[['AAPL', 'MSFT', 'NVDA']]
ma20 = close.rolling(20).mean()  # Compute all stocks at once

Data Quality & Processing

Handling Missing Values

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')

# Check missing values
missing_ratio = close.isna().sum() / len(close)
print(f"Missing-value ratio:\n{missing_ratio[missing_ratio > 0.1]}")

# Method 1: Forward fill (recommended)
close_filled = close.ffill()

# Method 2: Remove stocks with missing values
close_clean = close.dropna(axis=1, how='any')

# Method 3: Remove dates with missing values
close_clean = close.dropna(axis=0, how='any')

Handling Outliers

Reverse-split artifacts and bad prints show up as implausible daily moves. Mask them before computing signals:

close = data.get('price:adj_close')

# Daily moves beyond +/-50% are almost always data artifacts
daily_return = close.pct_change()
outlier = daily_return.abs() > 0.5

# Remove the affected dates
close_filtered = close[~outlier]

Common Data Table Reference

The examples below use the U.S. market (data.set_market('us')).

Price Data

# Adjusted close price (accounting for dividends and stock splits)
adj_close = data.get('price:adj_close')

# Raw close price
raw_close = data.get('price:close')

# Volume (shares)
volume = data.get('price:volume')

# Market value (USD)
market_value = data.get('etl:us_market_value')

Financial Statement Data

Quarterly statements are aligned to each SEC filing date.

# Diluted earnings per share
eps = data.get('us_income_statement:eps_diluted')

# Return on equity
net_income = data.get('us_income_statement:net_income')
equity = data.get('us_balance_sheet:total_stockholders_equity')
roe = net_income / equity

# Operating margin
revenue = data.get('us_income_statement:revenue')
operating_margin = data.get('us_income_statement:operating_income') / revenue

# Revenue year-over-year growth
rev_yoy = revenue.pct_change(4)

Taiwan-Only Data

Monthly revenue, institutional investor trading and margin trading data exist for the Taiwan market only. Find their dataset keys with data.search() under data.set_market('tw').

Save Data to CSV

close = data.get('price:adj_close')

# Save the entire DataFrame
close.to_csv('close_prices.csv')

# Save specific stocks
close[['AAPL', 'MSFT']].to_csv('selected_stocks.csv')

Load Data from CSV

import pandas as pd

# Load CSV
close = pd.read_csv('close_prices.csv', index_col=0, parse_dates=True)

# Convert to FinlabDataFrame
from finlab.dataframe import FinlabDataFrame

close = FinlabDataFrame(close)

# Now you can use FinLab methods
ma20 = close.average(20)