.resampledata() skips data
-
@backtrader seems we have discussed this before and not sure if this is an issue with the code, or an issue with my memory...
I've mentioned I occasionally see the system skipping or not showing the most recent data when using
backfill_from=
.With the following example code, I see the following scenarios:
-
Using
.resampledata()
to "register" the feed, withbackfill_from=
using the local static data, the output I see from the bar print innext()
shows every other bar. -
Using
.resampledata()
withoutbackfill_from=
never gets to a LIVE data status with the IB connection. -
Using
.adddata()
shows the expected output.
The code may be a bit more complicated than needed, but I have been using it to track down why I am not seeing valid indicator output when running live vs. in backtest (which may be related)
#!/usr/bin/env python # -*- coding: utf-8; py-indent-offset:4 -*- from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime as dt import pytz # Import the backtrader platform import backtrader as bt import backtrader.feeds as btfeed import backtrader.indicators as btind EST = pytz.timezone('US/Eastern') class TestDVStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function for this strategy''' dt = dt or self.data_spy.datetime.datetime(tz=EST) lstr = 'TEST: {}: {}'.format(dt.isoformat(), txt) print(lstr) def __init__(self): # To keep track of pending orders and buy price/commission self.trend = None self.datastatus = False self.data_spy = self.getdatabyname('SPY') # Add a MovingAverageSimple indicator based from SPY self.sma = btind.SMA(self.data_spy.close, period=200) def notify_data(self, data, status, *args, **kwargs): print('*' * 3, '%s DATA NOTIF: %s' % (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'), data._getstatusname(status)), *args) if status == data.LIVE: self.datastatus = True def notify_store(self, msg, *args, **kwargs): print('*' * 3, '%s STORE NOTIF: %s' % (dt.datetime.now().strftime('%Y-%m-%dT%H:%M:%S'), msg)) def next(self): if len(self) < 4: return if self.sma[-1] > self.sma[-2] > self.sma[-3] > self.sma[-4]: self.trend = 'up' else: self.trend = 'down' print('-- %004d' % len(self), str(self.datetime.datetime())) for i, d in enumerate(di for di in self.datas if len(di)): out = ['Data%d' % i, d._name, '%004d' % len(d), str(d.datetime.datetime()), str(d.close[0])] print('\t'.join(out)) self.log('TEST: %.2f SMA: %.3f TREND: %s' % (self.data_spy.close[0], self.sma[0], self.trend)) def runstrategy(): # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestDVStrategy) # Parse static SPY data file bfdata0 = btfeed.GenericCSVData(dataname='./datas/SPY-1D.csv', dtformat=('%Y-%m-%d'), datetime=0, high=1, low=2, open=3, close=4, volume=5, openinterest=6, sessionstart=dt.time(9, 30), sessionend=dt.time(16, 0), tz=EST) storekwargs = dict( host='127.0.4.1', port=7465, clientId=110, _debug=False ) ibstore = bt.stores.IBStore(**storekwargs) broker = ibstore.getbroker() cerebro.setbroker(broker) # SPY Live data timeframe resampled to 1 Day data0 = ibstore.getdata(dataname='SPY-STK-SMART-USD', # backfill_from=bfdata0, timeframe=bt.TimeFrame.Days, compression=1, sessionstart=dt.time(9, 30), sessionend=dt.time(16, 0), tz='EST5EDT') # cerebro.resampledata(data0, name='SPY', # timeframe=bt.TimeFrame.Days, compression=1) cerebro.adddata(data0, name='SPY') cerebro.run(runonce=False, tradehistory=True, exactbars=0) if __name__ == '__main__': runstrategy()
-
-
Worth looking into it.
- Using .adddata() shows the expected output.
This statement is probably not meant so. Because without
resampledata
, the feed returned by theIBStore
will download data inDays
(because thetimeframe
indicates it to do so), but it will later deliver theticks
.It would be much better to pass
bt.TimeFrame.Ticks
(which is what the feed actually delivers) and letresampledata
do its thing to download the historical data inbt.TimeFrame.Days
This is an area which would probably benefit from some rework (you already mentioned it and it is done with the
VisualChart
live feed, for other reasons):- Having
getdata
(actuallyIBData
) to directly return a resampled version of itself for the giventimeframe
What would also require:
- To separate the historical (for backfilling) download from the internals of
IBData
In order to be able to do a 2nd
resampledata
(and a 3rd if needed be) with a different (larger, smaller)timeframe/compression
combination.Some extra thought should have gone into the aforementioned bullet points at the time of development.
-
I somehow think the
backfill_from=
is part of the issue here. Also, not sure from your comments but note that in this script, I am only requesting one feed from IB.The issue is seen when I backfill static data and for some reason, when using the
.resampledata()
method, the system is dropping every other bar. This is identical to another issue we were seeing when I was trying to.resampledata()
from the InfluxDB feed. In that case,.adddata()
produced all bars..resampledata()
dropped every other bar. -
@RandyT said in .resampledata() skips data:
The issue is seen when I backfill static data and for some reason, when using the .resampledata() method, the system is dropping every other bar. This is identical to another issue we were seeing when I was trying to .resampledata() from the InfluxDB feed. In that case, .adddata() produced all bars. .resampledata() dropped every other bar.
This issue was due to not having specified the
sessionend
for the data, which collided with the algorithm in charge of knowing when the day has come to an end. This was also the reason to add the defaultsessionend
to, at least, all CSV data feeds, which has uncovered that some users were even not actually specifying the incoming timeframe, which was confusing the resampling code.