Anyone using Backtrader for live trading

[skytrading54]

Hi All,
Given that backtrader has live trading capability, wander if anyone has/is used/using it for live trading (more so with Interactive Brokers)...

[remroc]

Mostly paper trading for the moment... As I am still learning the fwk...

[backtrader]

During development, real orders were issued from backtrader to Interactive Brokers TWS and a maximum of 2 simultaneous data feeds were put in play.

As with anything which you may use to trade money with, you have to be aware of:

• No matter how much effort is put into keeping backtrader bug free: bugs may be there.

• Your own code may contain bugs (it doesn't have to throw an exception, it may simply not do what you are expecting it to do)

• Connectivity issues

  • From your internet connection
  • From IB

• TWS will exit every 24 hours. You would have to use one of the 3rd party solutions which are able to restart the platform. backtrader will try to backfill to recover (in most cases the re-start should happen outside of market trading hours, unless we consider the 24x7 nature of Forex)

The best approach would be to set up a test system with a paper trading account (they are associated to your main account to benefit from the same data permissions, but this apparently requires running from the same machine) and test long your scenarios before trying to do it live

[skytrading54]

Thanks @backtrader & @remroc,
I have been playing with backtrader & ibtest.py recently and found some below potential issues/enhancements

1. CFD's are not supported in backtrader fully, for getting market data / historical data they behave like cash, so need to modify the ibstore.py to include CFD like CASH as given below
   ''' if contract.m_secType == 'CASH': '''
   if contract.m_secType == 'CASH' or contract.m_secType == 'CFD':
   @backtrader - could you please consider including above in next release.

2. While strategies can be triggered preferably based on real time bars (it could be sourced as rtbar or made up from the market data), while putting orders it is essential to know the bid and ask price as well before putting live order. I couldn't find any way to get both of them in the call to next(), any ideas on
   a) how to get the mix of rtbar and market data.
   b) within market data should get all (bid, ask and trade (if IB is giving based on security type)), c) how to get all of this in next() and be able to distinguish the fields.

[backtrader]

Do you have a sample of the potential full notation needed for a CFD?. For example:

• Multipliers?
• Expiration/Strike Dates?
• Long/Short?

Actually none of the above may apply at all. This is a sample from another platform:

• IBUS500-SMART-CFD-USD. This seems a lot more like if a CFD is treated like a common stock and not as a CASH product as you mentioned above.

Another example:

• IBDE30-CFD-SMART-EUR-BAAVG

Where it is fully unknown what BAAVG means and where it is supposed to fit.

Bid/Ask Data

The platform delivers OHLC bars and bid/ask data is not considered (except for CASH products which don't offer anything else and the BID price is followed according to industry practices).

What IB calls Real Time Bars is a 5-second snapshot of the market activity and with it comes not bid/ask data.

Keeping a record of the bid/ask prices would be possible, but it will have to be implemented.

[skytrading54]

IB CFD's:
2 types, STOCK CFD - price is that of underlying stock, and INDEX CFD - price is derived by that of underlying index's immediate future contract.

Multipliers - not applicable.
Expiration/Strike Dates - not applicable,
Long/Short - both are possible.

IB don't give data for stock CFD, we need to source it for underlying stock.
Index CFD's which is relevant following works
IBDE30-CFD-SMART
IBUS500-CFD-SMART

[skytrading54]

Trying out on paper trading, found that while it recovers the open positions at start of program, it does not recover the open orders.

[backtrader]

Because the position has a general meaning for an asset and the orders may have been issued by a different client (manually over TWS for example)

Your algorithm may be ready to go short if you have a long position, but an open order may be a stop-loss which you have issued as a safety belt for failures in the algorithm (or the platform sustaining it)

It may be a good use case to selectively enable order download.

[backtrader]

@skytrading54

Long/Short was a badly formulated question in the sense that it tried to ask if, like with Options, you can have a different end symbol if you go for a PUT or a CALL.

It seems like the data-source vs trading-asset paradigm which can be used in the VisualChart data feed is applicable to this case too.

[skytrading54]

Let me reword it.
Using backtrader for IB paper trading (with simple strategy to buy / sell on particular conditions). This is a failover scenario.

1. A limit order is sent to exchange and accepted and got FILLED from this client. For some reason the program crashed, on restart backtrader download the position and I know I got a open position while making decisions - this is fine.

2. A limit order is sent to exchange and accepted (but NOT filled) from this client. For some reason the program crashed, on restart it has no clue of the existing open order at exchange as it does not download the orders (like what it does for positions).

There is an api reqOpenOrder which return open orders for that client, can we please look to add this to TODO list.

[rastegarr]

I'm currently testing Backtrader as the platform for our day-trading startup. I have a difficult time to connect to 100+ symbols realtime. The client gets connected and disconnected many times before all the symbols are added. Is there anybody with experience with that many symbols in backtrader?

[backtrader]

@rastegarr For sure not many people try to use those many symbols regardless of the platform.

One thing to remember is that TWS limits the amount of historical requests that can be issued in a certain amount of time.

From IB - Historical Data Limitations

Pacing Violations

All of the API technologies support historical data requests. However, requesting the same historical data in a short period of time can cause extra load on the backend and subsequently cause pacing violations. The error code and message that indicates a pacing violation is:

162 - Historical Market Data Service error message: Historical data request pacing violation

The following conditions can cause a pacing violation:

• Making identical historical data requests within 15 seconds;
• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.
• Also, observe the following limitation when requesting historical data:

Do not make more than 60 historical data requests in any ten-minute period.
If the whatToShow parameter in reqHistoricalData() is set to BID_ASK, then this counts as two requests and we will call BID and ASK historical data separately.

Unless you manually disable backfill_start (initial backfilling) and backfill (after each diconnection) by setting both to False during data feed creation you are continously going over this limit:

Do not make more than 60 historical data requests in any ten-minute period.

At least over that one. Some others could kick in. That limit will apply to any platform connecting to TWS or IBGateway.

Of course by disabling backfilling, the indicators will take longer to produce usable values.

[skytrading54]

@rastegarr
I am also trying to change code for multiple symbols however got issues in handling strategies next() call. Its call for each bar and couldn't identify for which of data0/1/2... it is called so that rest could be ignored in that call. This is causing multiple signals on same products.
Could you please share something on handling multiple symbols end to end if your have faced similar issue and workaround...

[ab_trader]

@skytrading54

Check out this link - https://www.backtrader.com/blog/posts/2015-09-03-multidata-strategy/multidata-strategy.html. Maybe it will help.

To issue the order for particular dataN the BUY order code is self.buy(data=dataN).

[backtrader]

@skytrading54 next is not called for individual data feeds.

It is called when the next unit of time of any of the data feeds has elapsed.

Scenarios

• A single data feed:

  Straightforward. next is called for each bar of the data feed

• B multiple data feeds aligned in time

  Straightfoward too. The same principle as above

• C multiple data feeds which are not aligned in time

  One of the data feeds may advance in time whilst the other(s) may remain static. This can happen, for example, if assets from different exchanges are in the system and the exchanges have different trading calendars. An example SPY and DAX. Bank holidays in the USA and Germany are not aligned (some are but most aren't).

  When the SPY is not trading because it's Thanksgiving Day, the DAX is happily trading. In this case next will be called and the following will be true:

  • The SPY data feed will not increase its length and checking the values at [0] will return the last known values
  • The DAX data feed will increase its length and the values at [0] will be the current ones
  • The Strategy will increase its length because one of the data feeds has moved forward (and value, cash and other things may have changed)

  You may check this which shows an extreme example in which several trading days were removed from the source:

  • https://www.backtrader.com/blog/posts/2016-09-17-data-synchronization/data-synchronization.html

Scenario C is the one applicable to multiple live data feeds, because some may not deliver data for a given period, whilst others will do.

The smaller the period, the higher the chances that some data feeds will not have data for a given period, whilst others (those with more liquidity) will keep pushing the clock forward.

To understand which data feeds have moved forward use: len(dataX) and see if it has increased with regards to the last time (you may also check dataX.datetime[0] or dataX.datetime.datetime(), but this will change in a replay scenario, with the length remaining constant, hence the recommendation to use len)

[backtrader]

Initial support for CFD products has been added.

The default request for any CFD security type will be for BID instead of TRADES

[skytrading54]

Thanks Backtrader,
I have taken this an ran the newer version but seems its not processing correctly, some obervations
python ibtest_12Jan17.py --port 8099 --clientId 2 --data0 IBUS500-CFD-SMART --timeframe Seconds --compression 5 --broker --rtbar

1. Historical data request is not seen going through..
   ***** STORE NOTIF: <error id=16777217, errorCode=420, errorMsg=Invalid Real-time Query:No historical market data for IBUS500/CFD@UKBBO Last 0>
2. next is not getting called even when real time data is coming through as seen in debug mode.

this is irrespective of --rtbar or not. so far I am not altering anything, will try to run it in trace to see what is happening.

[backtrader]

If the error states: Invalid Real-time Query, it seems difficult next may be called at all, because the code believes there is no incoming data.

That's the reason for the questions earlier in this thread, with regards to for example how to treat the -BAAVG string which for example Sierra Chart uses. See here

May it also be that the currency specifier is missing?

[skytrading54]

The issues in processing the data received for CFD's.

Basically the error is its processing the market data returned for CFD's.
CFD's we request only BID and IB provides it in callback tickPrice()
The current implementation of tickprice checks if its CASH then only it process, ignore rest.
tickerId = msg.tickerId
if self.iscash[tickerId]:
........

I have made the to include if its CFD's as well.
main change is
tickerId = msg.tickerId
if self.iscash[tickerId] or self.iscfd[tickerId]:
if msg.field == 1: # Bid Price
........
rest of the changes are to create/push/pop self.iscfd dictionary.
All the changes are in ibstore.py

I had forked a copy from dev branch and my changes are checked it in the GitHub
https://github.com/skytrading54/backtrader

somehow couldn't create a pull request under
https://github.com/skytrading54/backtrader

Please take a look to merge in the next cycle.

It now works as expected ....
python ibtest_12Jan17.py --port 8099 --clientId 1 --data0 IBHK50-CFD-SMART --timeframe Minutes --compression 1
Data0, 0215, 736342.252778, 2017-01-13T06:04:00.000000, 22953.87, 22954.87, 22952.86, 22953.87, -1.0, 0, 22950.47
Data0, 0216, 736342.253472, 2017-01-13T06:05:00.000000, 22953.87, 22957.87, 22952.86, 22957.87, -1.0, 0, 22952.47
Data0, 0217, 736342.254167, 2017-01-13T06:06:00.000000, 22957.87, 22964.87, 22955.86, 22963.87, -1.0, 0, 22955.67
Data0, 0218, 736342.254861, 2017-01-13T06:07:00.000000, 22963.87, 22963.87, 22960.86, 22960.87, -1.0, 0, 22958.07
***** DATA NOTIF: LIVE
Data0, 0219, 736342.255556, 2017-01-13T06:08:00.000000, 22960.87, 22961.87, 22959.87, 22960.87, 0.0, 0, 22959.47

[skytrading54]

typos ...

1. Basically the error is its NOT processing the market data returned for CFD's.
2. somehow couldn't create a pull request under
   https://github.com/mementum/backtrader