Anyone using Backtrader for live trading

skytrading54

created the pull request in mementum/backtrader

Changes in ibstore.py to get CFD market data. #240

you may include in next release after review.

backtrader

@skytrading54 It seems it's all about managing a CFD product, like if it were a CASH product. Is that all?

skytrading54

@backtrader yes that's right.

skytrading54

I am facing a delay issue in processing updates for IB when using multiple products having different market timings (meaning for some of them market is open and for some its not.)

eg: IBAU200, USO, IBUS100, Emini future.
while running this during US market hours while for 3 of them has market open but not Australia, the historical data is downloaded first 3 very quickly for 4th one is stuck somewhere and so the next in strategy is happening very slowly it takes up to 10 minutes to have next completed for historical data 1 minute compression/timeframe. Same is applicable if I run this Australia timing as US is not open by then.

If I remove IBAU200 (and add many other US products) it works fine (all next calls are done for historical data within few seconds). Wander what could be the issue.

I tried passing session start / session end time while creating each of the data feed, its still the same issue.

It seems something to with backtrader being waiting for response for first real time update of IBAU200 so that it can subscribe for historical data for the same which does not happen and so there is some sort of looping.

Any idea how this could be fixed ... if resorted this will help run one program across multiple market...

backtrader

This is an aspect which is being worked on (worked on as in right now) There are several factors in play here.

backtrader grew up from being purely meant for backtesting into adding datas which can keep on pumping. Things grew on top of the existing code.
Most of the initial use cases had to deal with ticks and seconds timeframes, being minutes a very large timeframe apparently.

There is suddenly, so to say, a 2nd generation of use cases which all involve large timeframes, waiting for the end of day to kick in to issue an order.

The problem arises because the larger timeframe goes live a lot sooner than the smaller timeframe, and the larger timeframe tries to wait for more incoming data to see if it can deliver. This makes sense in the case of replay, because the larger timeframe even if not complete can deliver and waiting seems (actually seemed) like an acceptable compromise.

But if the larger timeframe is being resampled it no longer makes sense to wait, because it will only be complete when it is complete (sessionend, timeframe/compression boundary, ...) and not any minute before.

backtrader

Let's not forget to offer a solution to the problem at hand. Use the parameter qcheck for the larger timeframes.

qcheck=0.05 for example (the default is 0.5 which is half a second) will reduce the lag during the historical shootout by a factor of 10.

Of course, the internal loop checking for a resampled data to deliver will tick faster (this will invisible to the end user, but keeps the CPU a lot more often active. Unless you CPU is really limited, it should make a huge difference to start with)

backtrader

And ... the development branch contains code to alleviate the situation. It will not only work during the initial backfilling stage, but during any stage, i.e.: if not all feeds have the LIVE status, there will no waiting, because some other feeds can (or will when the download is complete) produce data from historical sources.

backtrader

Surprisingly, in any case, the new code works against the faux data which is delivered even during the weekends, but not against a regular account, which would get only historical data now ...

skytrading54

Thanks backtrader will check with qchek parameter on the weekday when data is available in paper account and will take dev branch once done.

backtrader

The development branch may still undergo a couple of iterations. The new code works, but has the side effect of spinning up the CPU because there is no waiting time. A couple of extra use cases will make it.

Checking the qcheck effect can be done with the demo, which runs continuously even during the weekend.

newtrader

@backtrader I just recently started using backtrader (this is such a great learning experience) for backtesting. I wanted to use my strategy to place orders on a Bitcoin exchange. I have a pipe producing the data and the backtesting runs successfully using that same data.

But, where exactly should I place the code to make the calls to brokers api?

if self.dataclose[0] < self.dataclose[-1]:
# current close less than previous close

                if self.dataclose[-1] < self.dataclose[-2]:
                    # previous close less than the previous close

                    # BUY, BUY, BUY!!! (with default parameters)
                    self.log('BUY CREATE, %.2f' % self.dataclose[0])

                    # Keep track of the created order to avoid a 2nd order
                    self.order = self.buy() # Should this be replaced by e.g self.order = my_api.sell() ???

Kind Regards
NT

backtrader

You have to tell the system to which broker to connect. The API in the strategy doesn't change (or else it wouldn't make sense)

See for example Docs - Interactive Brokers

The same concepts apply to the other two live brokers, but basically withe the Store pattern (which is preferred)

...
store = bt.stores.IBStore(**myparams*)  # like port to connect to
broker = store.getbroker(**brokerargs)  # if any is needed
cerebro.broker = broker
...

You probably also want to use a data from the same service with store.getdata(dataname=xxx, **otherargs)

newtrader

@backtrader Thank you for the quick reply! I did see the this page where it leverages IBBroker as a store. But as I am trying to connect to a Exchange say Bitfinex, is there documentation i can create for custom Broker?

backtrader

There isn't a documentation for creating a broker because each broker has a different way of doing things. For example: Interactive Brokers has bracket orders (A group of 3 orders in which 2 of them bracket the 3rd in the middle) whereas Oanda has bracketing support (A single order with 3 possible prices, in which 2 of them bracket the price in the middle)

Some approaches were tried along the way and the best approach in any case would be:

Not to create a specific Order object even if subclassed from Order
Use a standard Order object and create broker specific order dicts, lists and other things inside the broker code
Use the Store pattern to avoid direct instantiation of specific Broker and Data classes. This also helps to keep common parameters (like for example host for the connection) unified
Use the code from Oanda as a basis.

In general you will need:

Background threads to process data and broker events
Synchronized queues to received the events from the background threads
A mapping between the data events (like not being able to access a data feed due to permissions -> NOTSUBSCRIBED) to the events defined in the backtrader classes
A mapping also for some broker events. Some brokers report expired orders as cancelled. It may not really be that important at the end, but if it's important for you, you should be able to discriminate.

Squirrel Singing

@backtrader Been using Backtrader to do simulated trading on Interactive Brokers and realize something unique about the last bar of delayed data. The closing price of the last bar of delayed data (the bar just before the first live data) to be made available even before its time period is up? Is that intentional?

stevetree

I use Backtrader in combination with IB. I went live to a real account 6 month ago and trade 8 future-strategies, all day and night, and I could say that you can rely on Backtrader. It is a quite reliable element in my production-chain. So, many many thanks to @backtrader ! You did a great work! Greeting from Munich/Germany.

Squirrel Singing

@stevetree , thanks for your response! Not too sure about the details of your strategy, but i find that the final delayed data bar is generated at the time of connection instead of at the appropriate time, eg 08:05:00 if i were to resample to 5 minute timeframe.

This is impacting all my indicators that are calculated based on delayed data bars.

Using the below screenshot as an example, GBPUSD closing price of 1.28984 at 8:05:00 is incorrect as it shows GBPUSD's closing price at the time of connection instead of at 08:05:00

A sample code is attached below if you wish to replicate what i showed in the screenshot.

import backtrader as bt
import datetime as dt
import pytz as pytz

class Test_Strategy (bt.Strategy):    
    def next (self):
        self.log (self.data.close[0])
    
    def notify_data(self, data, status):
        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status))
    
    def log (self, text):
        print (self.data.datetime.datetime(0),": ", text)

if __name__ == '__main__':
    ibstore = bt.stores.IBStore (host = '127.0.0.1', clientId = 110, port = 7497)
    data = ibstore.getdata (dataname = 'GBP.USD-CASH-IDEALPRO',       #GBP.USD-CASH-IDEALPRO, BARC-STK-LSE-GBP
                            rtbar = False,
                            tz=pytz.timezone('US/Eastern'),
                            )
    cerebro = bt.Cerebro()
    cerebro.broker = ibstore.getbroker()                                                          ###
    cerebro.resampledata (data, timeframe = bt.TimeFrame.Minutes, compression = 5, rightedge = True, boundoff = 0)
    cerebro.addstrategy (Test_Strategy)
    cerebro.run()

Squirrel Singing

@stevetree or other fellow traders, if you have the time, could you kindly take a look at my query?

Basically, the final delayed data bar is generated at the time of connection rather than at the time of bar complete formation. This is affecting the calculation of my indicators, such as Moving Average, as the final delayed data bar is used to arrive at the indicator value.

Has anyone noticed this?

If yes, does anyone have a solution?

For details on the code, pls refer to the earlier post.

Any response will be greatly appreciated!

Squirrel Singing

Just want to follow up on this issue that I face and wonder whether someone can kindly help me?

Thank you