market-datasets

name: market-datasets description: Fetch market data, fundamentals, and financial datasets from multiple sources (Stooq, NBP, Yahoo Finance, FRED, Tiingo, CCXT, FinancialData.Net) with intelligent routing, plus long-history construction patterns for investment research. Use when users request stock prices, indices, FX rates, economic indicators, cryptocurrency data, fundamentals, options, financial statements, or need to build multi-decade return series for backtesting. argument-hint: "[ticker] [start-date] [end-date]" allowed-tools: Bash, Read

Market Datasets

Unified market data fetching with intelligent source selection, plus long-history asset construction for investment research.

When to Use

• Fetching: Historical stock prices, indices, FX rates, economic indicators, crypto
• Construction: Building multi-decade return series for backtesting (ETF + academic backfill)
• Comparison: Cross-source data validation
• Conversion: ISIN/ticker lookups across sources

Supported Sources

Dividend Treatment

Key insight: Stooq Close = Yahoo adj_close (validated 95.8% match rate). See references/dividend_treatment.md.

Core Workflow

Step 1: Determine Approach

Step 2: Fetch Data

CLI (all scripts are self-contained with PEP 723 inline dependencies):

uv run scripts/fetch_unified.py PKO 2024-01-01 2024-12-31
uv run scripts/fetch_stooq.py WIG20 2024-01-01
uv run scripts/fetch_nbp.py USD 2024-01-01 2024-12-31
uv run scripts/fetch_yahoo.py AAPL 2024-01-01
uv run scripts/fetch_financialdata.py AAPL 2024-01-01 2024-12-31

Python:

import sys; sys.path.append('market-datasets/scripts')
from fetch_unified import fetch_market_data

df = fetch_market_data('pko', start_date='2024-01-01')        # Polish → Stooq
df = fetch_market_data('AAPL', start_date='2024-01-01')       # US → Yahoo
df = fetch_market_data('USD', source='nbp', table='A')        # FX → NBP
df = fetch_market_data('BTC/USDT', start_date='2024-01-01')   # Crypto → CCXT
df = fetch_market_data('PLXTRDM00011', start_date='2024-01-01')  # ISIN → auto-resolve

Step 3: Handle Results

All fetchers return pandas DataFrames with standardized columns:

Economic data (FRED, NBP): Date, Value, Currency/Series_ID.

Quick Examples

# 1. Simple fetch with auto-routing
df = fetch_market_data('pko', start_date='2024-01-01', end_date='2024-12-31')

# 2. Multi-source comparison
from fetch_unified import UnifiedMarketDataFetcher
fetcher = UnifiedMarketDataFetcher()
comparison = fetcher.compare_sources('pko', ['stooq', 'yahoo'], start_date='2024-01-01')

# 3. ISIN batch fetch
from fetch_yahoo_direct import fetch_by_isin
prices, dividends, splits, mapping = fetch_by_isin(
    ['PLXTRDM00011', 'US0378331005'], start_date='2024-01-01'
)

For all usage patterns, see references/usage_patterns.md.

Long-History Asset Construction

For investment research requiring multi-decade return series (backtesting, risk calibration), use the ETF-primary, academic-backfill pattern.

Strategy

ETF/fund data (real, 1976-2025)  ←  PRIMARY: highest quality
    ↕ splice point
Academic data (synthetic, 1928+)  ←  BACKFILL: extends history

Standard Asset Class Proxies

Key Construction Patterns

Splice: Use primary where available, backfill for earlier dates. No blending.

splice_date = primary.index.min()
backfill_part = backfill[backfill.index < splice_date]
result = pd.concat([backfill_part, primary]).sort_index()

Synthetic bond return from yield:

coupon = yield_series.shift(1) / 12
price_change = -duration * (yield_series - yield_series.shift(1))
monthly_return = coupon + price_change

Total return from price + dividends:

total_return = (price + dividend / 12) / price.shift(1) - 1

For complete source details, download patterns, and validation methodology, see references/long_history_construction.md.

Reference implementation: research/risk-profile-calibration/build_asset_data_v2.py.

Ticker Formats

Full reference: references/ticker_formats.md.

Caching

File-based caching enabled by default:

• Location: data/cache/market_data/{source}/
• Default TTL: 24 hours

df = fetch_market_data('pko', use_cache=False)           # Disable cache
fetcher = UnifiedMarketDataFetcher(cache_hours=48)       # Custom TTL

Configuration

API Keys

See references/api_setup.md.

Error Handling

try:
    df = fetch_market_data(ticker)
except ValueError as e:
    print(f"Invalid ticker or no data: {e}")  # Bad ticker, empty result, date issues
except Exception as e:
    print(f"Network or other error: {e}")

Best Practices

1. Use auto-routing by default — let the fetcher choose the best source
2. Enable caching — reduces API load, enables offline work
3. Validate results — check not df.empty, expected columns, no NaN gaps
4. Batch requests — loop with fetcher instance, don't re-instantiate
5. Respect rate limits — 1-2s delay between requests (built in)

Reference Documentation

Load these when needed for detailed information: