Next() is skipping data?

cwse

Hi there!

Why is backtrader only analysing its next() statement over a subset of my datas date range?
I have imported many ticker dataframes into datas, restricting their date ranges from 2012-01-01 to 2016-12-31, and verified this date range is setting correctly on my dataframes prior to adding them into cerebro. However backtrader’s first next() iteration starts at 2012-11-26… for no apparent reason!
I have no SMAs or other date restrictions in place. Any ideas what could be causing this?
My next() code which shows this is happening is:

def next(self): 
    for i, d in enumerate(d for d in self.datas if len(d)):
        print('date: {}'.format(d.datetime.date(0)))

FYI my data is added as follows:


class CustomDataLoader(btfeeds.PandasData):
    lines = ('TOTAL_SCORE', 'Beta',)
    params = (
        ('openinterest', None),  # None= column not present
        ('TOTAL_SCORE', -1),
        ('Beta', -1))
    datafields = btfeeds.PandasData.datafields + (['TOTAL_SCORE', 'Beta'])

…


for ticker in Index_Constituents[IndexDates[3]]:
    datarange = Stocks[ticker].loc[
        (Stocks[ticker].index>=datetime.datetime.strptime(args.fromdate, "%Y-%m-%d")) &
        (Stocks[ticker].index<datetime.datetime.strptime(args.todate, "%Y-%m-%d"))
    ]
    data = CustomDataLoader(dataname=datarange)
    cerebro.adddata(data, name=ticker)

Thank you,
cwe

backtrader

No idea with no no data. A long shot: some of the data feeds do not start as early as expected. timeframe is also not being set and it is unclear which actual timeframes are being used, which can confuse datetime synchronization.

cwse

Good point. Which dataframe does backtrader choose as the 'master' in terms of the date range? I assumed it would take the largest dataframe. Perhaps I can set a stock Index as the daterange some how?

Also what is timeframe and how can I use it?

Thanks again!

backtrader

Which dataframe does backtrader choose as the 'master' in terms of the date range?

None. But the minimum period of any lines object in the framework is 1. It will for example the chose period for a SimpleMovingAverage. The Strategy will be in prenext until the minimum period is met for all the lines objects.

If a data feed starts 11 months after others, the minimum period will be first met then.

timeframe is a parameter to data feeds to indicate which one the data has.

cwse

Thank you @backtrader, found the problem..! One dataframe starts 11 months later as you suspected. How can I work around this? I still want to analyse the data from before this point in time. Is there any way to override /work around this?

backtrader

It's in the answer above. Override prenext

cwse

Hi @backtrader, there is no prenext() docs page, and I have checked through the following docs to try and find something on this..:

Strategy
Quickstart
Platform concepts
Operating the Platform
Cerebro

It would be greatly appreciated if you could give me some more info which would assist in solving this, instead of stating it has to do with prenext().

Thank you

backtrader

From Docs - Strategy

...

Childhood: prenext
indicators declared during conception will have put constraints on how long the strategy needs to mature: this is called the minimum period. Above init created a SimpleMovingAverage with a period=15.
As long as the system has seen less than 15 bars, prenext will be called (there is a default method which is a no-op)
...

Further below in the Strategy Reference

...
prenext()
This method will be called before the minimum period of all datas/indicators have been meet for the strategy to start executing
...

Cerebro also documents it Docs - Cerebro

...

Call the strategies’ next method to let the strategy evaluate the new data (and maybe issue orders which are queued in the broker)
Depending on the stage it may be prenext or nextstart before the minimum period requirements of the strategy/indicators are met
Internally the strategies will also kick the observers, indicators, analyzers and other active elements
...

In Docs - Operating the platform

...
But because they are not regular iterators, two additional methods exist:

prenext
Called before the minimum period for the line iteratorhas been met. nextstart Called exactly ONCE when the minimum period for the line iterator has been met.
The default behavior is to forward the call to next, but can of course be overriden if needed.
...

And in several samples, one that specifically deals with data feeds which deliver at different times

Sample - data-multitimeframe

cwse

Thanks @backtrader, yes I understand the definition of prenext() as your quotes above show.. but they don't show how to use it to manage differing timeframes :-)

The sample code looks like it is useful though! It appears all I need after all is this?!

    def prenext(self):
        self.next()