By name: debug-performance description: > Diagnoses slow QuantConnect Python (.py) and C# (.cs) backtests using the Performance Chart first and Python cProfile only when the chart does not pinpoint the hot function. Trigger phrases: "slow backtest", "high CPU", "algorithm slow", "CPU usage", "RAM usage", "memory usage", "performance chart", "profiling", "bottleneck", "debug performance", "taking too long", "optimize algorithm".
/debug-performance - QuantConnect Performance Debugging
Invoked on a slow backtest. Work top to bottom. Find the dominant cost before
changing code. Same Performance Chart rules for main.py and main.cs; Python
can add cProfile only when Section 4 says to.
0. Never add error-hiding patterns
Never introduce pytry/exceptcstry/catch or
pyhasattr/getattr/setattr/isinstancecsreflection. They hide the exact
failure or slow path. A profiler wrapper measures work; it is not a catch. If
existing code wraps the slow region in one, remove it, re-run, read the real
behavior, then continue.
1. Enable Performance Chart first
Record the original pyset_start_date/set_end_datecsSetStartDate/SetEndDate
and universe, then shrink both: usually 1 to 3 months and the smallest universe
that still reproduces the slowdown. Set
Settings.PerformanceSamplePeriod = TimeSpan.FromDays(7)
in Initialize, then run the short backtest.
Open the results and inspect the Performance Chart. Record the dominant series:
the tallest spike, or the sustained plateau if there is no single spike. Do not
optimize before naming the series.
2. Route by dominant series
Read which series dominates:
- Selection, Consolidators, OnData, Schedule, Subscriptions, Securities, Transactions, SplitsDividendsDelisting, Slice, HistoryDataPoints, or ActiveSecurities -> Section 3.
- CPU, ManagedRAM, or TotalRAM high with no single time-series dominant -> Section 4 for Python, or keep using the chart for C#.
- DataPoints low -> Section 3, Subscriptions row. If multiple series spike together, start with the earliest causal source: Subscriptions before Consolidators, Consolidators before OnData, Selection before Securities or Transactions. Treat CPU and RAM as symptoms until the chart or profiler names the code path.
3. Fix by bottleneck
Change only the code path that corresponds to the dominant series. Re-run the
same short backtest after each change and compare the same chart region.
For rolling calculations, use this order: built-in indicator first, indicator
extension second, Security.session for cached OHLCV/session data third, and
manual RollingWindow/deque only when LEAN has no built-in equivalent. Example:
a trailing mean of closes is an SMA, so use pyself.sma(security, period) instead
of storing closes and averaging them manually.
| Series | Meaning | Fix |
|---|---|---|
| Selection | Universe add/remove and selection functions. | Reduce coarse universe size; move expensive fundamental queries out of the filter; cache results across calls. |
| Consolidators | Consolidation, indicator updates, consolidator handlers. | Reduce universe size; consolidate at a coarser resolution; share one consolidator where logic allows. |
| OnData | OnData and alpha update time. |
Move heavy logic to a scheduled event; replace manual series calculations with a built-in indicator or indicator extension before considering a manual window. |
| Schedule | Scheduled event handler time. | Reduce event frequency; cache intermediate results computed in the handler. |
| Subscriptions | Time reading subscribed data. | Reduce data resolution or universe size; confirm only needed resolutions are subscribed. |
| Securities | Security updates, security changes, symbol changes. | Reduce ActiveSecurities count, usually fewer holdings or open orders. |
| Transactions | Order event processing. | Batch orders via portfolio targets; reduce rebalance frequency. |
| SplitsDividendsDelisting | Corporate action events. | Reduce universe size or move corporate-action logic out of the handler. |
| Slice | Time creating the Slice object. |
Reduce subscription count, resolution, and custom data fields before optimizing handlers. |
| HistoryDataPoints | History provider data points. | Reduce History calls; replace rolling calculations with indicators/extensions; use Security.Session for cached OHLCV/session data; request fewer symbols, fields, or bars. |
| ActiveSecurities | Selected securities plus holdings and open orders. | Prune stale symbols; liquidate or cancel old positions/orders before broadening the universe. |
Logging rules: measure with Log(...) in the narrow handler |
||
| being debugged. Never use the Object Store for profiler output. Never log inside | ||
OnData or an often-firing scheduled event without a counter limit. |
||
| Remove diagnostic logs after the fix. |
4. Python profiler with logs
Use cProfile only when CPU, ManagedRAM, or TotalRAM is high and the Performance Chart does not isolate the expensive function. This is Python only. C# does not have a built-in cProfile equivalent; use the chart series exclusively.
At the top of
main.py, before the class:In
OnEndOfAlgorithm, disable the profiler and log the top cumulative-time lines. Do not save the report to the Object Store:Read the cumulative time column in the backtest logs. The top function is the bottleneck; map it back to a Section 3 series and apply the matching fix. Remove the profiler after measurement unless the user explicitly wants another profiling run.
5. Checklist
- No banned error-hiding pattern (Section 0).
- Original dates + universe recorded before shrinking.
- Performance Chart enabled; short backtest run to expose the dominant series.
- Dominant series identified before applying any fix (Section 2).
- Fix targets the real bottleneck series, not a guess (Section 3).
- Built-in indicators/extensions considered before adding py
RollingWindow/dequecsRollingWindow. - Profiler used only for Python when chart alone does not pinpoint the function.
- Profiler output logged with
Log; Object Store not used. - Original period + universe restored; final backtest confirms the spike is gone.