q.data Roadmap

Only open work is listed here. Completed items are removed.

Design boundaries

QRT should model financial data independently of any execution engine. Generic objects must be usable with pandas/Parquet and other backtest engines before a LEAN serializer is involved.

• q.data owns normalized data models, transformations, quality reports, and point-in-time correctness.
• q.calendar owns exchange sessions, timestamp-to-session alignment, and trading-time grids.
• q.data.lean owns LEAN paths, ZIP members, scaling, CSV schemas, SIDs, and reference-database serialization.
• q.bt.lean owns Lean CLI processes and result artifacts; it consumes q.data.lean but does not duplicate its writers.
• Vendor adapters own retrieval and source-specific translation. Core models must not depend on a vendor’s ticker or corporate-action schema.

Canonical security identity

• Introduce a SecurityMaster or equivalent normalized table whose primary key is a stable source-independent security_id, not the current ticker.

  master = q.data.SecurityMaster.from_frame(
      securities,
      security_id="isin",
      ticker="ticker",
      listed_at="listed_at",
      delisted_at="delisted_at",
  )

• Define canonical fields for security type, share class, exchange, quote currency, display name, listing interval, and optional identifiers such as ISIN, FIGI, CUSIP, SEDOL, and vendor IDs.

• Represent ticker/vendor aliases as dated intervals. Lookups must support both ticker_at(security_id, timestamp) and security_at(ticker, exchange, timestamp) without applying today’s ticker to historical data.

• Validate overlapping ticker intervals, reused tickers, missing listing bounds, duplicate permanent IDs, changing share classes, and conflicting currencies/exchanges.

• Provide deterministic map-interval generation from alias history. The generic result should describe identity periods; q.data.lean should be the only layer that converts those periods to LEAN map-file conventions.

• Define serialization to canonical pandas and Parquet layouts with schema version metadata. Round trips must preserve nullable identifiers and timezone semantics.

Point-in-time universes

• Add a generic membership-event schema containing universe ID, permanent security ID, effective-from/effective-to timestamps, publication timestamp, weight, shares held, market value, and source metadata.

• Build session-indexed snapshots without survivorship leakage:

  snapshots = q.data.universe.snapshots(
      memberships,
      sessions=q.calendar.schedule(...),
      id_column="isin",
      effective_from="added_at",
      effective_to="removed_at",
  )

• Keep effective time and knowledge/publication time separate. Support an available_at policy so research can exclude a reconstitution that was not yet known at the simulated decision time.

• Support additions, removals, full replacements, unchanged carry-forward, explicitly empty universes, and weighted constituent updates.

• Validate duplicate members, overlapping membership intervals, unknown securities, weights outside policy, sums materially different from one, publication after use, and snapshots outside listing intervals.

• Return a normalized long table first. Dense matrices and engine-specific daily files are derived representations, not the source of truth.

• Add diff/audit operations that explain why a member entered, exited, or changed weight between two snapshots.

Corporate actions and adjustment factors

• Define canonical split and cash-dividend events keyed by permanent security ID. Preserve announcement, ex, record, and payable dates separately; factor construction should use the documented effective-date policy.

• Implement backward cumulative factor construction from raw daily closes:

  factors = q.data.corporate_actions.factors(
      prices=daily_prices,
      splits=splits,
      dividends=dividends,
      exchange="XSTO",
  )

• Resolve each event to the previous exchange session through q.calendar, match the raw reference close, and retain enough intermediate columns to audit every cumulative factor change.

• Support multiple actions on one effective date, reverse splits, currency metadata, action corrections/cancellations, and configurable numerical precision. Reject impossible split ratios and dividends that produce nonpositive adjustment factors.

• Add forward and backward adjustment operations for OHLCV with explicit modes: raw, split-adjusted, price-adjusted, and total-return. Volume/share adjustment policy must be explicit rather than inferred from the price mode.

• Reconstruct actions from adjacent factor rows as a validation step and compare split ratios/distributions with the input events.

• Keep taxes, withholding, rights, spin-offs, merger consideration, and stock dividends outside the first API unless their accounting semantics are modeled explicitly.

Session-aware market-data aggregation

• Add q.data.resample_trades for canonical trades using exchange intervals supplied by q.calendar:

  bars = q.data.resample_trades(
      trades,
      frequency="1min",
      exchange="XSTO",
      empty="omit",
  )

• Define open/high/low/close selection, quantity summation, timestamp label, interval closure, empty-bar behavior, and deterministic ordering for equal event timestamps.

• Aggregate daily bars by exchange session rather than UTC or local calendar date. Respect early closes, late opens, exchange breaks, and official auction prints according to an explicit inclusion policy.

• Add quote aggregation separately. Bid/ask OHLC, last size versus average size, one-sided updates, crossed quotes, and stale-side carry-forward need independent policies; quote bars must never be synthesized silently from trades.

• Provide bar reconciliation helpers that compare tick-to-minute, minute-to-hour, and intraday-to-daily results within configured tolerances.

Structured market-data validation

• Expand beyond exception-only schema checks with a structured quality report:

  report = q.data.validate_market_data(
      trades=trades,
      quotes=quotes,
      bars=bars,
      exchange="XSTO",
  )

• Each finding should carry a stable code, severity, symbol, timestamp or session, affected column, observed value, expected rule, and optional sample rows. Reports should support raise_for_errors(), summary tables, JSON, and machine-readable pipeline thresholds.

• Validate timestamp ordering, duplicate policy, timezone awareness, session membership, missing sessions, incomplete normal/half-day sessions, OHLC invariants, finite prices, nonnegative quantities, and symbol metadata.

• Add trade/quote checks for spread sign, zero or one-sided quotes, stale quotes, suspicious/corrected trades, exchange-condition mappings, and extreme price or size changes. Statistical anomaly checks must be warnings by default, not destructive cleaning.

• Add cross-resolution reconciliation and corporate-action checks: scaled price reversibility, aggregate equality, map/listing bounds, factor reference closes, reconstructed events, and universe membership inside listing dates.

• Separate validation from repair. Any repair operation must produce an audit table describing old value, new value, reason, and policy version.

LEAN data adapter

• Add an optional q.data.lean namespace that serializes canonical QRT objects into a user-selected LEAN workspace. Importing ordinary q.data must not require Lean CLI, Docker, or QuantConnect packages.

• Provide high-level writers with path-like destinations and dry-run plans:

  q.data.lean.write_equity(
      trades=trades,
      quotes=quotes,
      bars=bars,
      symbol="VOLV-B",
      market="sweden",
      root=workspace / "data",
  )
  q.data.lean.write_map_files(master, root=workspace / "data")
  q.data.lean.write_factor_files(factors, root=workspace / "data")
  q.data.lean.write_universe(snapshots, security_type="index", ...)

• Centralize LEAN’s 10,000x equity price scaling, invariant numeric formatting, lowercase directory conventions, date partitioning, ZIP/member naming, and resolution-specific trade/quote schemas.

• Support tick, second, minute, hour, and daily data without pretending LEAN automatically falls back between resolutions. Writers should state exactly which subscriptions/history calls the produced files satisfy.

• Generate map files from canonical identity intervals and factor files from canonical factors. Preserve stable permanent identity across ticker changes, listings, delistings, and ticker reuse.

• Implement broad daily, ETF, and Index constituent writers. SID generation must be deterministic from security type, market numeric ID, and first map date; changing any SID input should require an explicit migration/regeneration.

• Write or patch market-hours and symbol-properties databases atomically. Never silently overwrite unrelated custom entries. Surface the risk that Lean CLI database updates replace local files.

• Treat custom market name/ID as workspace configuration, detect collisions, and emit the required Python/C# registration snippet. Do not claim that data registration also supplies fee, fill, buying-power, settlement, or brokerage models.

• Add a validate_tree operation that inspects paths, ZIP members, headers, scaling, row schemas, sort order, map/factor coverage, metadata entries, universe SIDs, and requested resolutions without launching a backtest.

• Make writes idempotent and atomic. Support dry_run=True, explicit replacement scopes, checksums, and a manifest recording source schema, generator version, market ID, calendar version, and output counts.

• Add round-trip/read-back tests against LEAN’s readers or golden fixtures for XNYS and XSTO. The Sweden fixture under lean/demo-generated-data should become an integration test, not production API internals.

Synthetic market-data helpers

• Keep q.stats.random_walk as the price-path primitive and evaluate a separate q.data.synthetic namespace only after a second independent use case confirms the API.

• Candidate helpers include exchange-session intraday timestamp generation, configurable trade-size processes, and explicit quote generation from a supplied spread model:

  quotes = q.data.synthetic.quotes_from_trades(
      trades,
      spread=0.01,
  )

• Synthetic outputs must be visibly labeled, deterministic under a seed, schema-compatible with canonical data, and make no claim of realistic microstructure unless a calibrated model is supplied.

• Do not promote fixture-only constants, hard-coded symbols, or LEAN paths into generic APIs. Those belong in integration-test fixtures.

Delivery sequence

1. Stabilize security-master, membership-event, corporate-action, trade, quote, and quality-finding schemas.
2. Land the required q.calendar session-label and interval APIs.
3. Implement point-in-time universe snapshots and corporate-action factors.
4. Implement session-aware trade/quote aggregation and structured validation.
5. Build q.data.lean writers and tree validation over the generic models.
6. Convert the generated Sweden demo into golden and end-to-end tests.
7. Evaluate reusable synthetic-data helpers after the core contracts settle.

Acceptance criteria

• Every canonical transformation has deterministic schema, timezone, sorting, nullability, and error policies documented and tested.
• Point-in-time tests demonstrate that future ticker, constituent, and corporate-action information cannot leak into earlier snapshots.
• XSTO tests cover normal sessions, half days, holidays, DST, ticker changes, delistings, splits, dividends, and an index reconstitution.
• LEAN files generated from the golden fixture complete direct-data, universe, and SMA backtests with zero failed data requests.
• Generic q.data tests run without Lean CLI or Docker installed.

Ideas

These candidates fit q.data, but are not yet committed roadmap items.

• q.data.sources.fred — macroeconomic series with release-date and vintage-aware handling
• q.data.sources.fama_french — normalized factor datasets with explicit units and source metadata