Struggle to pull diffrent datas (Data Feeds - Multiple Timeframes, Data Feeds - Resample, Data Feeds - Replay)

[kevkev]

Hello, maybe you could help me with it..

I struggle to use diffrent data sources or access/resample and use the data feeds with diffrent time frames/periods. for example I'm using the tick data for price updating but I also I need to use "look back from the current price"-method that will get high&low of 24 periods of 30m to calculate and update ATR.
this is the script I'm using at the moment;

from future import (absolute_import, division, print_function, unicode_literals)
import sys, argparse
import os.path

import pandas as pd

import backtrader as bt
import datetime

import backtrader.analyzers as btanalyzers

from strategies import GoldenCross, BuyHold, KevStrategy

if name == 'main':

cerebro = bt.Cerebro(stdstats=False) 

cerebro.addstrategy(KevStrategy)
modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
datapath = os.path.join(modpath, '/Users//PycharmProjects/Algo_trading_binance/data_mining/BTCUSDT_trades_2022_03_300.csv')
# modpath1 = os.path.dirname(os.path.abspath(sys.argv[0]))
# datapath1 = os.path.join(modpath1, '/Users//PycharmProjects/Algo_trading_binance/data_mining/2022_30minutes_BTCUSDT.csv')

data = bt.feeds.GenericCSVData(dataname=datapath,
                               fromdate=datetime.datetime(2022, 3, 29),
                               todate=datetime.datetime(2022, 4, 1),
                               nullvalue=0.0,
                               dtformat='%Y-%m-%d %H:%M:%S.%f',
                               tmformat='%H:%M:%S:%f',
                               timeframe=bt.TimeFrame.Ticks,
                               # compression=1,
                               datetime=4, close=1, volume=2, openinterest=-1)

cerebro.adddata(data, name='data')

# call your Tick data by data.close[0] and your Minutes bar data by data1.close[0]
cerebro.replaydata(data, timeframe=bt.TimeFrame.Ticks, compression=1, name='data')  # stored in variable self.data OR self.data0
data.plotinfo.plotmaster = data 
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=30, name='data1')  # stored in variable self.data1 which is every 30-minute interval
#cerebro.resampledata(data2, timeframe=bt.TimeFrame.Minutes, name='mins')

# Set our desired cash start
cerebro.broker.setcash(300000.0)

# Add a FixedSize sizer according to the stake
# cerebro.addsizer(bt.sizers.FixedSize, stake=300)

# Set the commission - 0.1% ... divide by 100 to remove the %
cerebro.broker.setcommission(commission=0.00075)

print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.run()

print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.plot()

This is how the CSV ticks data is looks like:

id,price,qty,quoteQty,time,isBuyerMaker,isBestMatch
1310078677,47222.0,0.0016,75.5552,2022-03-30 21:34:10.004,False,True
1310078678,47222.0,0.00953,450.02566,2022-03-30 21:34:10.061,False,True
1310078679,47222.0,0.00119,56.19418,2022-03-30 21:34:10.061,False,True
1310078680,47222.0,0.00291,137.41602,2022-03-30 21:34:10.061,False,True
1310078681,47222.0,0.0006,28.3332,2022-03-30 21:34:10.118,False,True
1310078682,47221.99,0.0029,136.943771,2022-03-30 21:34:10.472,True,True
1310078683,47222.0,0.00099,46.74978,2022-03-30 21:34:10.490,False,True
1310078684,47222.0,0.00104,49.11088,2022-03-30 21:34:11.605,False,True
1310078685,47222.0,0.00399,188.41578,2022-03-30 21:34:11.608,False,True
1310078686,47222.0,0.004,188.888,2022-03-30 21:34:11.608,False,True
1310078687,47222.0,0.00059,27.86098,2022-03-30 21:34:11.608,False,True
1310078688,47222.0,0.00097,45.80534,2022-03-30 21:34:11.608,False,True

this is the ATR method called from the strategy:

    def next(self):
        range_total = 0
        for i in range(-23, 1):
            true_range = self.data1[0].high[i] - self.data1[0].low[i]
            range_total += true_range
        ATR = range_total / 24

        self.log(f'Close: {self.dataclose[0]:.2f}, ATR: {ATR:.4f}')

Will appreciate any help here as I couldn't find proper solution at the moment..

Thanks!

[Pierre Cilliers 0]

@kevkev Hi man.

If you have tested that your 30-minute interval high and low prices are correctly replayed, maybe try access the different prices as:

def next(self):
        range_total = 0
        for i in range(-24, 0):
            true_range = self.data1.high[i] - self.data1.low[i]
            range_total += true_range
        ATR = range_total / 24

The fact that you try self.data1[0].high[i], indicates that you are looking at the most recent data1 entry (self.data1[0]) and then trying to access historical prices which is not in this object but rather in the entire self.data1 object.

Alternatively, I found this to work great for my case:

If your would like to output of use all the 23 previous high and low prices, you could use the following:

high_prices = self.data1.high.get(size = period, ago=-1)
low_prices = self.data1.low.get(size = period, ago=-1)

where you will replace the period parameter with how many intervals back you want to analise. From here is will return something similar to this if your print(high_prices in your def next()):

high_prices = array('d', [1.22431, 1.22467, 1.22466, 1.22454, 1.22457]) # if the period param = 5 for instance

You can therefore then loop through the array and calculate what you need.

Just shout if you need any further assistance

[kevkev]

@kevkev said in Struggle to pull diffrent datas (Data Feeds - Multiple Timeframes, Data Feeds - Resample, Data Feeds - Replay):

price

Thanks @Pierre-Cilliers-0 for the help, The tick data/CSV.file that I'm using as in the example above provides only the tick prices and volume not HLOCV. So when I'm replay the data it is converted to 30m with HLOCV data ?

Also do you know a way to run the strategy and get the required data for the ATR method immediately and not to wait until having enough data(12 periods of 30m) and instead get this data before the initial start of the current prices taken from the tick data?

Also what will you offer if I have CSV data that is already have 30m bars with HLOCV. How can I access the 30m separate CSV file for updating the ATR and use the tick separate CSV for updating prices?
Example for the 30m bars:

Time,Open,High,Low,Close,Volume
2022-03-29 00:00:00,47122.21,47509.07,47050.0,47332.94,1354.15238
2022-03-29 00:30:00,47332.94,47606.01,47332.93,47397.88,857.87244
2022-03-29 01:00:00,47397.89,47450.75,47336.01,47412.71,431.91864
2022-03-29 01:30:00,47412.72,47600.0,47369.28,47537.04,682.73805
2022-03-29 02:00:00,47537.05,47589.0,47458.5,47580.66,519.74138
2022-03-29 02:30:00,47580.66,47596.5,47383.38,47408.56,638.86549
2022-03-29 03:00:00,47408.56,47423.82,47322.11,47381.7,385.69904
2022-03-29 03:30:00,47381.7,47424.37,47315.0,47335.06,371.33161
2022-03-29 04:00:00,47335.06,47655.54,47320.65,47587.91,667.76336

[Pierre Cilliers 0]

@kevkev Hi,

Yes, although your tick data is bid, ask and volume, the cerebro.replaydata actaully calculates OHLCV for you in a new data source. Unfortunately backtrader cannot calculate a "warm start", meaning that it runs from the first data entry. It HAS TO first calculate the technical indicator after 12 30-minute periods before running your strategy (meaning it has a "cold start").

Regarding an additional data source of 30-minute intervals, you can simply add the data via a new bt.feeds. Just make sure that you now actually assign names to the various data to ensure you are tracking and calling them correctly.
Just ensure that their time periods overlap correctly.

[kevkev]

@pierre-cilliers-0 Thanks again! it works with the replay data after I resample the data to 1 second bars of OLHCV. But it runs very slow when I'm replay the data and use it for diffrent indicator such the as the ATR... also I could not understand how can I run the data with diffrent CSV data files that are having diffrent timeframes for the same purpose.. maybe in this way it will run faster?

Maybe do you know how can I run large data feeds faster ? (much faster?)

[Pierre Cilliers 0]

@kevkev I would suggest to play around and read exactbars = True vs exactbars = False

Also encountering long runs

[kevkev]

@pierre-cilliers-0

The exactbars are working with the replay method ?