@pierre-cilliers-0

The exactbars are working with the replay method ?

Any thoughts ?

I use seconds CSV data feed OLHCL of one month and replay the data 3 times 1. for seconds(as is) 2. for 5 minutes 3. for 30 minutes and use it for diffrent indicators such as meager spikes, calculate ATR and the seconds bars are for the prices updating.

At the current way when I replay the data it takes more then 1.5 hour to run one back test. and I need to add more time intervals for diffrent indicators meaning that I will require to replay the data more then those 3 times and it means that it will take much more time..

Also I'm using only one month at the moment and would like to use more than 1 month for those tests.

Do you guys know any way to significant decrease the time for those runs ?

@pierre-cilliers-0 Thanks again! it works with the replay data after I resample the data to 1 second bars of OLHCV. But it runs very slow when I'm replay the data and use it for diffrent indicator such the as the ATR... also I could not understand how can I run the data with diffrent CSV data files that are having diffrent timeframes for the same purpose.. maybe in this way it will run faster?

Maybe do you know how can I run large data feeds faster ? (much faster?)

@tianjixuetu did you find any better way to speed up the back tests?

@kevkev said in Struggle to pull diffrent datas (Data Feeds - Multiple Timeframes, Data Feeds - Resample, Data Feeds - Replay):

price

Thanks @Pierre-Cilliers-0 for the help, The tick data/CSV.file that I'm using as in the example above provides only the tick prices and volume not HLOCV. So when I'm replay the data it is converted to 30m with HLOCV data ?

Also do you know a way to run the strategy and get the required data for the ATR method immediately and not to wait until having enough data(12 periods of 30m) and instead get this data before the initial start of the current prices taken from the tick data?

Also what will you offer if I have CSV data that is already have 30m bars with HLOCV. How can I access the 30m separate CSV file for updating the ATR and use the tick separate CSV for updating prices?
Example for the 30m bars:

Time,Open,High,Low,Close,Volume
2022-03-29 00:00:00,47122.21,47509.07,47050.0,47332.94,1354.15238
2022-03-29 00:30:00,47332.94,47606.01,47332.93,47397.88,857.87244
2022-03-29 01:00:00,47397.89,47450.75,47336.01,47412.71,431.91864
2022-03-29 01:30:00,47412.72,47600.0,47369.28,47537.04,682.73805
2022-03-29 02:00:00,47537.05,47589.0,47458.5,47580.66,519.74138
2022-03-29 02:30:00,47580.66,47596.5,47383.38,47408.56,638.86549
2022-03-29 03:00:00,47408.56,47423.82,47322.11,47381.7,385.69904
2022-03-29 03:30:00,47381.7,47424.37,47315.0,47335.06,371.33161
2022-03-29 04:00:00,47335.06,47655.54,47320.65,47587.91,667.76336

Hello, maybe you could help me with it..

I struggle to use diffrent data sources or access/resample and use the data feeds with diffrent time frames/periods. for example I'm using the tick data for price updating but I also I need to use "look back from the current price"-method that will get high&low of 24 periods of 30m to calculate and update ATR.
this is the script I'm using at the moment;

from future import (absolute_import, division, print_function, unicode_literals)
import sys, argparse
import os.path

import pandas as pd

import backtrader as bt
import datetime

import backtrader.analyzers as btanalyzers

from strategies import GoldenCross, BuyHold, KevStrategy

if name == 'main':

cerebro = bt.Cerebro(stdstats=False) 

cerebro.addstrategy(KevStrategy)
modpath = os.path.dirname(os.path.abspath(sys.argv[0]))
datapath = os.path.join(modpath, '/Users//PycharmProjects/Algo_trading_binance/data_mining/BTCUSDT_trades_2022_03_300.csv')
# modpath1 = os.path.dirname(os.path.abspath(sys.argv[0]))
# datapath1 = os.path.join(modpath1, '/Users//PycharmProjects/Algo_trading_binance/data_mining/2022_30minutes_BTCUSDT.csv')

data = bt.feeds.GenericCSVData(dataname=datapath,
                               fromdate=datetime.datetime(2022, 3, 29),
                               todate=datetime.datetime(2022, 4, 1),
                               nullvalue=0.0,
                               dtformat='%Y-%m-%d %H:%M:%S.%f',
                               tmformat='%H:%M:%S:%f',
                               timeframe=bt.TimeFrame.Ticks,
                               # compression=1,
                               datetime=4, close=1, volume=2, openinterest=-1)

cerebro.adddata(data, name='data')

# call your Tick data by data.close[0] and your Minutes bar data by data1.close[0]
cerebro.replaydata(data, timeframe=bt.TimeFrame.Ticks, compression=1, name='data')  # stored in variable self.data OR self.data0
data.plotinfo.plotmaster = data 
cerebro.replaydata(data, timeframe=bt.TimeFrame.Minutes, compression=30, name='data1')  # stored in variable self.data1 which is every 30-minute interval
#cerebro.resampledata(data2, timeframe=bt.TimeFrame.Minutes, name='mins')

# Set our desired cash start
cerebro.broker.setcash(300000.0)

# Add a FixedSize sizer according to the stake
# cerebro.addsizer(bt.sizers.FixedSize, stake=300)

# Set the commission - 0.1% ... divide by 100 to remove the %
cerebro.broker.setcommission(commission=0.00075)

print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.run()

print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.plot()

This is how the CSV ticks data is looks like:

id,price,qty,quoteQty,time,isBuyerMaker,isBestMatch
1310078677,47222.0,0.0016,75.5552,2022-03-30 21:34:10.004,False,True
1310078678,47222.0,0.00953,450.02566,2022-03-30 21:34:10.061,False,True
1310078679,47222.0,0.00119,56.19418,2022-03-30 21:34:10.061,False,True
1310078680,47222.0,0.00291,137.41602,2022-03-30 21:34:10.061,False,True
1310078681,47222.0,0.0006,28.3332,2022-03-30 21:34:10.118,False,True
1310078682,47221.99,0.0029,136.943771,2022-03-30 21:34:10.472,True,True
1310078683,47222.0,0.00099,46.74978,2022-03-30 21:34:10.490,False,True
1310078684,47222.0,0.00104,49.11088,2022-03-30 21:34:11.605,False,True
1310078685,47222.0,0.00399,188.41578,2022-03-30 21:34:11.608,False,True
1310078686,47222.0,0.004,188.888,2022-03-30 21:34:11.608,False,True
1310078687,47222.0,0.00059,27.86098,2022-03-30 21:34:11.608,False,True
1310078688,47222.0,0.00097,45.80534,2022-03-30 21:34:11.608,False,True

this is the ATR method called from the strategy:

    def next(self):
        range_total = 0
        for i in range(-23, 1):
            true_range = self.data1[0].high[i] - self.data1[0].low[i]
            range_total += true_range
        ATR = range_total / 24

        self.log(f'Close: {self.dataclose[0]:.2f}, ATR: {ATR:.4f}')

Will appreciate any help here as I couldn't find proper solution at the moment..

Thanks!