Backfill periods

monoid

Good day, and firstly thank you very much to the developers of Backtrader for the hours you have put in.

I have a question concerning the implementation of backfilling of data, specifically the initial backfill before LIVE data are processed for IB. The relevant documentation is:
https://www.backtrader.com/docu/live/ib/ib.html#ibdata-feeds

Where "maximum possible duration" implicitly references:
https://interactivebrokers.github.io/tws-api/historical_limitations.html

For my use case, many times I do not need to backfill 1800s of data, 60s is sufficient. This can be determined in fact from my strategies _minperiod, so my question is: why the greedy backfill? Can we not be more frugal and take only what we need? Was this done because the correct _minperiod is not accessible to the feed?

Code ref example:
After _minperiod was set on the strategies, cerebro calls:
https://github.com/backtrader/backtrader/blob/master/backtrader/cerebro.py#L1538
which goes to
https://github.com/backtrader/backtrader/blob/master/backtrader/feeds/ibdata.py#L457
which goes to
https://github.com/backtrader/backtrader/blob/master/backtrader/stores/ibstore.py#L673

It seems from that 2nd link like maybe I could hack this in by setting my IBData objects datetime to:
now() - datetime(minperiod*timeframe*compression)
or something like that? Sorry if that code is offensive :')

And thanks again.

backtrader

@monoid said in Backfill periods:

why the greedy backfill?

It's not greedy. It's what's offered in a single request and it takes basically the same amount of time to get the greedy data as to get less.

@monoid said in Backfill periods:

Was this done because the correct _minperiod is not accessible to the feed?

It's done because it's what's fits in a single request. It has no other implications. Yes, big data has for sure made Trump win the election and the brits to vote for Brexit, but it didn't have any influence here.

@monoid said in Backfill periods:

Can we not be more frugal and take only what we need?

Specify fromdate to the data feed and less will be taken.

monoid

@backtrader said in Backfill periods:

it takes basically the same amount of time to get the greedy data as to get less

Today yes, but I worry what will happen with net neutrality being repealed. Jokes aside, in my case the data were taking maybe 50ms to process each datapoint but:
a) I realise that was probably an issue with the processing of the received data not the downloading of that data and
b) that issue seems to have magically disappeared today and the 1800s is no longer a bottleneck.

Thank you for your reply.