Skip to content

FinLab Data Download: Use data.get for U.S. Prices and Fundamentals

Use data.get() with a table:column key to download FinLab datasets. If you do not know the dataset key, start with data.search(); if you only need an index or a sector, filter with data.us_universe().

Goal Start with Example
Switch to U.S. data data.set_market() data.set_market('us')
Download prices, volume, fundamentals, or revenue data.get() data.get('price:adj_close')
Find available dataset keys data.search() data.search('revenue', market='us')
Limit the index or sector universe data.us_universe() with data.us_universe(index='SP500'):

Next steps: first-time users should read Getting Started. For full function signatures, see the finlab.data API reference.

Access permissions and free coverage

VIP receives the latest published data; daily prices normally cover the previous trading day. Other datasets follow their own update schedules. U.S. data and open reference tables have their own coverage, shown in the catalog. Free Taiwan historical data ends on 2023-12-31 (last trading day: 2023-12-29).

VIP only: the Taiwan convertible-bond datasets cb_price, cb_published_info, and cb_converted_status. After the trial ends, downloading these datasets requires a subscription; programs using them will receive a permission error.

Use data.search('cb', details=True) to inspect auth (free / vip) and free_until (inclusive free cutoff). The default data.search() still returns dataset keys.

Downloading Data

Use the data.get function to download over 10 years of historical records for thousands of stocks at once:

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')
date AAPL ... TSLA
2015-01-02 27.33 ... 14.62
... ... ... ...
2023-05-02 169.59 ... 161.83 
from finlab import data

data.set_market('tw')
close = data.get('price:close')

Taiwan prices use the same layout, with Taiwan stock IDs as columns. price:open, price:high, price:low, price:close, and price:volume are English keys for Taiwan daily prices.

The return type is FinlabDataFrame (an extended Pandas DataFrame). If you are familiar with Pandas, you can get started right away; beginners may want to read the Pandas 10-minute guide first.

  • Vertical axis (rows): closing prices on market trading dates.
  • Horizontal axis (columns): stock symbols, making it very convenient for building stock screening strategies.

How to skip the login prompt?

You can log in programmatically (examples below) to bypass the GUI authentication.

What other data is available?

Visit the Database Catalog to find available datasets and their key names.

Free data has a shorter range

Free users can download most datasets for backtesting, but recent data is not included. Upgrade to VIP to access the latest data.

Use the AI Assistant to find data

Not sure about the dataset name? After installing the FinLab Skill in your AI coding assistant, simply ask "What revenue-related datasets are available?" to get the correct data.get() parameters and usage examples.

Download concurrency and memory

FINLAB_MAX_DOWNLOAD_WORKERS limits Python download threads for data.get() and data.gets(). It defaults to 8 and must be a positive integer. Set it to 1 to download sequentially in the calling thread when your environment limits threads. The setting is read at download time; Pyodide keeps its existing download behavior.

import os
from finlab import data

os.environ['FINLAB_MAX_DOWNLOAD_WORKERS'] = '1'
data.set_market('us')
close, volume = data.gets('price:adj_close', 'price:volume')

del close, volume  # Remove your references when the frames are no longer needed.
data.free_memory()

data.free_memory() releases data and category caches in RAM, keeping FileStorage files, timestamps and current settings. Later reads can reload from disk. CacheStorage has no disk backup and must download again after release. DataFrames you still reference remain alive, and Python may not immediately return memory to the operating system.

Search Available Fields (data.search)

Use data.search(keyword=None, market=None) to search for available field names. The return format is table:column.

  • keyword: keyword (case-insensitive); pass None to list all
  • market: defaults to the active market (data.set_market()), or 'tw' when none is set; use 'us' (U.S. stocks), 'us_fund' (U.S. ETFs and funds), 'tw' (Taiwan stocks), or 'all' (every market)
  • An invalid market value raises a ValueError
from finlab import data

# U.S. market keyword search
close_us = data.search('close', market='us')

# U.S. revenue and earnings fields
revenue_us = data.search('revenue', market='us')

# List all U.S. market fields
all_us = data.search(market='us')

# Search all markets
price_all = data.search('price', market='all')

Separate date partitions when updating datasets daily

data.search(..., details=True) returns dataset, auth, free_until, and a boolean requires_date_range column. True means data.get() requires both start and end. False means you can call data.get(key) directly, subject to authentication, data permissions, and download quota.

catalog = data.search('', market='tw', details=True)  # Use market='us' for U.S. data
whole_tables = catalog.loc[~catalog['requires_date_range'], 'dataset']

# Update ordinary datasets using your existing FileStorage configuration
for key in whole_tables:
    data.get(key)

# Request date partitions separately, choosing a symbol and published dates
ticks = data.get('tw_tick:2330', start='2026-06-01', end='2026-06-01')

tw_tick:2330, tw_minute:2330, and tw_futures_minute:TX_202610 are examples of three date-partitioned families, not a complete symbol directory. Replace the stock or contract and dates as needed; see intraday data. This flag describes required date arguments, separately from data frequency: intraday_trading contains daily data, while tw_price_60min is a downloadable wide table of 60-minute bars. Both have False; do not exclude them by name or frequency.

Catalog source: Firestore data_categories/finlab_tw_stock (Taiwan) and data_categories/finlab_us_stock (US), with LRU caching.

Automatic Login to Retrieve Historical Data

Please log in before downloading. Jupyter/Colab, local Python, and VSCode can all use the browser-based login flow:

import finlab

finlab.login()

A more secure approach

For headless or automated environments, log in locally first and run python -m finlab token --env, then set the printed FINLAB_REFRESH_TOKEN, FINLAB_SESSION_ID, and FINLAB_API_KEY values as environment variables. Legacy FINLAB_API_TOKEN and finlab.login('token') still work for compatibility, but do not write tokens directly in notebooks or scripts.

Open the command prompt and enter: 

set FINLAB_REFRESH_TOKEN=<YOUR_REFRESH_TOKEN>
set FINLAB_SESSION_ID=<YOUR_SESSION_ID>
set FINLAB_API_KEY=<YOUR_API_KEY>

Open a terminal: 

export FINLAB_REFRESH_TOKEN=<YOUR_REFRESH_TOKEN>
export FINLAB_SESSION_ID=<YOUR_SESSION_ID>
export FINLAB_API_KEY=<YOUR_API_KEY>

Market and Sector Scope

Use data.us_universe to limit the data scope to an index, a sector, an industry, an exchange, or a security type. For example, to get the closing prices of Technology stocks in the S&P 500:

from finlab import data

data.set_market('us')
with data.us_universe(index='SP500', sector='Technology'):
    close = data.get('price:adj_close')

Every data.get() call inside the with block returns only the selected columns; the previous universe is restored when the block exits.

Parameter Values Description
index 'SP500' / 'SPX', 'NASDAQ100' / 'NDX' Latest index constituents
sector e.g. 'Technology' Sector name(s), regex matching
industry e.g. 'Semiconductors' Industry name(s), regex matching
exchange 'NASDAQ', 'NYSE', 'AMEX' Listing exchange(s)
security_type e.g. 'common_stock' Security type(s); 'common_stock' gives clean U.S. equities

U.S. sectors: Technology, Healthcare, Financial Services, Consumer Cyclical, Industrials, Communication Services, Consumer Defensive, Energy, Basic Materials, Real Estate, Utilities. The values come from data.get('us_company_profile') (columns sector, industry, exchange, security_type).

Index filters use the latest constituents

index='SP500' selects today's constituents for the entire price history. That is convenient for examples, but a historical backtest on it has survivorship bias. For point-in-time membership, see the historical index backtest in finlab.data.

Exact sector matching

sector and industry use regular expressions, so a pattern matches every name that contains it. Anchor the pattern to match one name exactly, for example sector='^Energy$'.

A filter that matches no securities (for example a misspelled sector) raises a ValueError and leaves the previous universe unchanged. Use data.reset_universe() to remove a universe set with data.set_universe().

Taiwan market scope (Taiwan market only)

Under the default Taiwan market, data.universe() filters by exchange and Taiwan industry:

Market Code Description
ALL All markets, including TSE, OTC, Emerging, and public offerings
TSE Listed stocks (Taiwan Stock Exchange)
OTC OTC stocks (Taipei Exchange)
TSE_OTC Listed + OTC stocks
ETF Exchange-Traded Funds

ETF-related categories:

Category Description
domestic_etf ETFs with Taiwan stocks as constituents
foreign_etf ETFs with foreign assets as constituents
leveraged_etf Leveraged ETFs
vanilla_futures_etf Non-leveraged futures ETFs
leveraged_futures_etf Leveraged futures ETFs 
from finlab import data

data.set_market('tw')
# Listed and OTC stocks, excluding the ETF category
with data.universe(market='TSE_OTC', exclude_category=['ETF']):
    close = data.get('price:close')

The category (include) and exclude_category (exclude) filters take Taiwan industry names, which are the Chinese names of the Taiwan industry classification in the security_categories dataset. Both use regular-expression matching; when both are set, the system first selects the category matches, then removes the exclude_category matches.

FinlabDataFrame Operations

data.get() returns a FinlabDataFrame which extends Pandas DataFrame with all standard Pandas operations plus built-in accessors for stock selection.

Basic Pandas Operations

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')
rev = data.get('us_income_statement:revenue')  # quarterly

# Arithmetic
ret = close / close.shift() - 1  # daily returns
rev_yoy = rev / rev.shift(4) - 1  # quarterly revenue YoY growth

# Rolling calculations
ma20 = close.average(20)  # 20-day moving average (FinLab extension)
std20 = close.rolling(20).std()  # 20-day volatility

# Comparison → boolean DataFrame
cond = close > ma20  # price above MA
golden_cross = ma20 > close.average(60)  # golden cross

# Logical combination
position = cond & golden_cross  # both conditions
position = cond | golden_cross  # either condition

Ranking & Selection

eps = data.get('us_income_statement:eps_diluted')
raw_close = data.get('price:close')  # unadjusted price for valuation
pe = raw_close / eps.rolling(4).sum()  # trailing-12-month P/E
pe = pe[pe > 0]  # drop loss-making companies

position = pe.is_smallest(10)  # lowest 10 PE stocks
position = pe.is_largest(10)  # highest 10 PE stocks
ranked = pe.rank(axis=1, pct=True)  # per-date percentile rank

Cross-Sectional Factor Processing (.cs)

df.cs (cross-sectional) operates across all stocks on each date. Useful for factor ranking and standardization:

net_income = data.get('us_income_statement:net_income')
equity = data.get('us_balance_sheet:total_stockholders_equity')
roe = net_income / equity  # quarterly ROE

roe.cs.rank()  # per-date percentile rank
roe.cs.zscore()  # per-date standardization (mean 0, std 1)
roe.cs.winsorize(0.05, 0.95)  # per-date percentile winsorization
roe.cs.bucket(5)  # per-date quintile bucketing
roe.cs.demean()  # per-date de-mean

# Select top 20% ROE stocks
position = roe.cs.rank() > 0.8

Within-Industry Analysis (.sector)

df.sector computes within the same sector, useful for sector-neutral analysis. Pass a symbol-to-sector mapping for U.S. stocks; without a mapping, .sector uses the Taiwan industry classification:

profile = data.get('us_company_profile')
sectors = dict(zip(profile['symbol'], profile['sector']))

# Transform: same methods as .cs, but within sector
roe.sector(sectors).rank()  # within-sector rank
roe.sector(
    sectors
).zscore()  # within-sector standardization (about the same as neutralization)

# Aggregate: sector-level statistics
roe.sector(sectors).mean()  # sector average ROE
roe.sector(sectors).std()  # sector ROE standard deviation
roe.sector(sectors).median()  # sector median

# Select top 30% ROE within each sector
position = roe.sector(sectors).rank() > 0.7

Portfolio Weight Construction (.weight)

df.weight builds refined weights from selection results:

position = roe.cs.rank() > 0.8  # select stocks first

# Chain weight construction
weights = (
    position.weight.inverse_volatility(
        window=60, price=close
    ).weight.limit_turnover(  # inverse-volatility weighting
        max_turnover=0.3
    )  # cap turnover at 30%
)

Pass price=close for U.S. stocks: without it, inverse_volatility reads Taiwan adjusted prices. weight.cap_industry() caps weights per Taiwan industry category.

See finlab.dataframe reference for all methods.

Parallel Batch Download

Need faster downloads? data.gets() supports parallel batch downloads and returns regular FinlabDataFrame objects.

Common Errors and Solutions

Error 1: DatasetNotFoundError - Dataset Not Found

Symptom: data.get() raises finlab.exceptions.DatasetNotFoundError (a RuntimeError subclass) that lists the closest valid keys

from finlab.exceptions import DatasetNotFoundError

data.set_market('us')
key = 'price:adj_closing'  # Wrong: incorrect field name
try:
    close = data.get(key)
except DatasetNotFoundError as e:
    print(e)  # the message lists the closest valid keys, such as us_price:adj_close

Causes: - Dataset name is misspelled (the correct name is adj_close, not adj_closing) - The dataset does not exist in the database - Login has not been completed or the login state is invalid, preventing data access

Solution:

from finlab import data
from finlab.exceptions import DatasetNotFoundError

# Method 1: Search for the correct field name first
matching_fields = data.search('close', market='us')
print(matching_fields)

# Method 2: Use try-except to handle the error
data.set_market('us')
try:
    close = data.get('price:adj_close')
    print(f"Data downloaded successfully, range: {close.index[0]} ~ {close.index[-1]}")
except DatasetNotFoundError as e:
    print(f"Dataset not found: {e}")
    print("Use data.search('keyword') to find the correct field name")
    print(
        "Or visit https://studio.finlab.finance/en/database to browse available datasets"
    )
except Exception as e:
    print(f"Download failed: {e}")
    print("Please check:")
    print("1. Have you run finlab.login() successfully?")
    print("2. Is the network connection working?")
    print("3. Do you have permission to access this dataset?")

Error 2: Empty DataFrame

Symptom: Download succeeds but the DataFrame has no data

close = data.get('price:adj_close')
print(close)
# Empty DataFrame
# Columns: []
# Index: []

Causes: - data.truncate_start / data.truncate_end exclude every date of the dataset - Free user's data is outside the available range - The data source temporarily has no data

A universe filter that matches nothing (for example a misspelled sector) does not return empty data: it raises a ValueError.

Solution:

from finlab import data

data.set_market('us')

# A misspelled sector raises instead of returning empty data
try:
    with data.us_universe(sector='Technolgy'):
        close = data.get('price:adj_close')
except ValueError as e:
    print(e)
    profile = data.get('us_company_profile')
    print("Available sectors:", sorted(profile['sector'].dropna().unique()))

with data.us_universe(sector='Technology'):
    close = data.get('price:adj_close')

# Check data completeness
if close.empty:
    print("Warning: downloaded data is empty")
    print("Possible reasons:")
    print("1. truncate_start / truncate_end exclude every date")
    print("2. The dataset itself has no data")
elif close.shape[0] < 10:
    print(f"Warning: too few data rows (only {close.shape[0]})")
else:
    print(f"Data is normal: {close.shape[0]} trading days, {close.shape[1]} stocks")

Error 3: Network Timeout

Symptom: No response for a long time when downloading, eventually throws Timeout or connection error

close = data.get('price:adj_close')
# finlab.exceptions.SignedDownloadReadTimeout: Download of us_price:adj_close failed (connection timed out). Check your network, proxy or firewall, then retry data.get().

Causes: - Unstable network connection - Data volume is too large (e.g., full market 10+ years of data) - Server temporarily unresponsive

Solution:

from finlab import data
from finlab.exceptions import DataError
import time

data.set_market('us')
try:
    close = data.get('price:adj_close')
except DataError as e:
    if getattr(e, "kind", None) == "timeout":
        print("Network timeout, attempting retry...")
        time.sleep(5)  # Wait 5 seconds
        try:
            close = data.get('price:adj_close')
            print("Retry successful")
        except Exception as retry_error:
            print(f"Retry failed: {retry_error}")
            print("Suggestions:")
            print("1. Check network connection")
            print("2. Try again later")
            raise
    else:
        raise

# finlab automatically caches downloaded data; the second run reads from cache

Error 4: Login or Permission Failure

Symptom: Unable to log in, cached credentials expired, or permission error when downloading data.

import finlab

finlab.login()
# Complete the browser login flow

Causes: - Browser login has not completed, or local credentials expired - Headless environment variables FINLAB_REFRESH_TOKEN, FINLAB_SESSION_ID, and FINLAB_API_KEY are incomplete - Insufficient account permissions (free user accessing VIP-only data)

Solution:

import finlab

# Method 1: Log in again (recommended)
try:
    finlab.login()
    print("Login successful")

    # Verify data access
    from finlab import data

    data.set_market('us')
    close = data.get('price:adj_close')
    print(f"Data access is working, latest date: {close.index[-1]}")

except Exception as e:
    print(f"Login failed: {e}")
    print("\nPlease check:")
    print("1. Did the browser login complete?")
    print("2. Is the account a valid VIP member?")
    print(
        "3. Are FINLAB_REFRESH_TOKEN / FINLAB_SESSION_ID / FINLAB_API_KEY set in headless environments?"
    )
    print("4. Is the network connection working?")

# Method 2: Check membership level (determine if latest data is accessible)
from finlab import data

close = data.get('price:adj_close')  # get_role() needs at least one data.get() call
if not data.is_vip():
    print(f"Current role: {data.get_role()}; free accounts only receive older data")
    print("Upgrade to VIP for the latest data: https://finlab.finance/en/payment")

Reference Resources

Structured download progress

data.get() and data.gets() accept progress='display' | 'event' | 'silent'. Agents and workers should use event or silent mode to avoid creating Jupyter HTML display objects.

events = []
data.set_market("us")
close, volume = data.gets(
    "price:adj_close",
    "price:volume",
    progress="event",
    progress_callback=events.append,
)

The event schema is {phase, dataset?, completed?, total?, cache_hit?, download_bytes?, elapsed_ms, timestamp}.