Using buffered indicator data - broker.get_value() 'nan'

[kriku]

I thought that the following could be of interest for someone.

I try to develop a swing trading strategy. By definition, it is all about good entry and exit points. For every individual stock, there could be any number of entry points but scanning hundreds of them results in a reasonably stable flow.

Entry point rules are based on both indicators of the single stock and the whole bunch of them. For instance, the strategy does not enter a trade if the average price of the sector is falling.

As hundreds of lines are scanned and only 2-3 positions maintained simultaneously, it is not known how many trades could be made with any single stock. However, I would like to know it. Cutting the list down to the stock in question is not a solution because it would disable the criteria basing on the average price movements of the whole list.

I tried to be smart and cached the sector data on disk and loaded it as separate lines:

class BufferedSectorIndicatorsCSVData(bt.CSVDataBase):

    lines = ('datetime', 'sector_mean', 'hull300_slope', 'sma_slope',)

    def _loadline(self, linetokens):
        i = itertools.count(0)
        date_txt = linetokens[next(i)]
        time_txt = linetokens[next(i)]

        dtxt = date_txt + " " + time_txt

        y = int(dtxt[0:4])
        m = int(dtxt[5:7])
        d = int(dtxt[8:10])
        h = int(dtxt[11:13])
        minute = int(dtxt[14:16])

        dt = datetime(year=y, month=m, day=d, hour=h, minute=minute)

        dtnum = bt.date2num(dt)

        self.lines.datetime[0] = dtnum
        self.lines.sector_mean[0] = float(linetokens[next(i)])
        self.lines.hull300_slope[0] = float(linetokens[next(i)])
        self.lines.sma_slope[0] = float(linetokens[next(i)])

        return True

However, this fails in quite an interesting way. Everything works but broker.get_value() starts to return 'nan' even for the stock that has correct price data. It can be observed in different ways; I checked it in the Sizer with the debugger and saw that all data.tick_* fields were correctly filled in and the data.p.dataname pointed to the real stock data (that I was not accidentally trying to "buy" a cached indicator data). Therefore the issue is not exactly similar to the one discussed here: https://community.backtrader.com/topic/1373/why-cerebro-broker-getvalue-is-nan

The solution was to add fictive OLCH data to the buffered lines:

        self.lines.open[0] = 0
        self.lines.close[0] = 0
        self.lines.high[0] = 0
        self.lines.low[0] = 0
        self.lines.volume[0] = 0
        self.lines.openinterest[0] = 0

Is this a bug or a feature?

Are there better / more backtraderish ways to use cached data?

[tianjixuetu]

I just met this question,and I find it is the NaN in my data,when I fillna(method="ffill"),it becomes normal.

for file in file_list:
    data = pd.read_csv("C:/data/future/day/"+file)
    datetime_list=list(data['datetime'])
    begin_date=datetime_list[0]
    if begin_date>"2010-01-01":
        end_date=datetime_list[-1]
        df1=df[df['datetime']>=begin_date]
        df1=df1[df1['datetime']<=end_date]
        if len(df1)!=len(data):
            print(file)
            df1=df1[['datetime']]
            df1.index=range(len(df1))
            # df2=pd.concat([df1,data],join="outer",keys="datetime",axis=1)
            df2=pd.merge(df1,data,on='datetime',how='outer')
            df2=df2.fillna(method="ffill")
            df2.to_csv("C:/data/future/day/"+file,index=False)