For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Anyone use backtrader to do live trading on Bitcoin exchange?



  • @sfkiwi yes, that issue was reported some time ago and there are 2 PRs about it: https://github.com/bartosh/backtrader/pull/13 and https://github.com/bartosh/backtrader/pull/6

    Unfortunately they're both outdated and require rebase. Can you check if any of them fixes the issue for you?



  • @ed-bartosh

    Currently backfill_from is not supported. I'll be happy to see this implemented. I hope it's not that hard to implement considering working example from ibdata.py.

    will add it to my backlog developments :)

    can you maybe open the option for 'issues' in your repo. I have a couple more ideas for developments and maybe other people also would like to participate. ccxt topic is very active and using this one super long thread is messy



  • @jacob > can you maybe open the option for 'issues' in your repo.

    done



  • Has anyone managed to fix sync/ gathering partial candle data issue mentioned above?
    If the solution is here https://github.com/bartosh/backtrader/pull/6, why cannot we merge ? (github noob here, sorry :p). Btw, thank you @Ed-Bartosh for this fabulous work :)



  • Hi @ed-bartosh!

    I tried to run on bitmex exchange live:

    data = bt.feeds.CCXT(exchange='bitmex', symbol='BTC/USD', timeframe=bt.TimeFrame.Ticks, compression=1)

    When I tried to run live using Bitmex exchange I got some errors:
    0_1535874007208_Screen Shot 2018-09-02 at 04.37.15.png

    I open a issue (#20) and fixed on my repo fork.

    Cloud you accept a pull request? My user is @dedeco on github. If not, the fix is on issue description.

    Thanks!!



  • @dedeco > Cloud you accept a pull request?

    Absolutely.

    BTW, You don't need to ask permissions for this, just submit PR. In worse case scenario it'll be rejected :)



  • @totoro said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    if the solution is here https://github.com/bartosh/backtrader/pull/6, why cannot we merge ?

    I believe comments in that PR explain why it's not merged.

    Btw, thank you @Ed-Bartosh for this fabulous work :)

    You're welcome. It's actually a community work, not just mine. Many people contributed to it.
    BTW, you have chance to do it too :)



  • Hello all, and thanks @Ed-Bartosh and others for your phenomenal work.

    I am having a problem trading using backtrader on the Bittrex exchange (I have not tried other exchanges yet). The code below, slightly modified from Soren Pallesen's example 01-09-2018, places a limit order far from the current price to simply check the trading functionality:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    import sys
    import time
    from datetime import datetime, timedelta
    
    import backtrader as bt
    import ccxt
    
    class TestStrategy(bt.Strategy):
    
        def start(self):
            self.counter = 0
            print('START')
    
        def prenext(self):
            self.counter += 1
            print('prenext len %d - counter %d' % (len(self), self.counter))
    
        def __init__(self):
                self.dmi = bt.indicators.DMI(self.data, period=3)
                self.order = None
    
        def next(self):
            print('------ next len %d - counter %d' % (len(self), self.counter))
    
            print(self.dmi.plusDI[0], self.dmi.minusDI[0], self.dmi.adx[0])
    
            self.counter += 1
    
            print('*' * 5, 'NEXT:', bt.num2date(self.data0.datetime[0]),
                  self.data0._name, self.data0.open[0], self.data0.high[0],
                  self.data0.low[0], self.data0.close[0], self.data0.volume[0],
                  bt.TimeFrame.getname(self.data0._timeframe), len(self.data0))
    
            if not self.getposition(data) and self.order==None:
                print('\nBUYING')
                self.order = self.buy(data, exectype=bt.Order.Limit, size=0.5,
                                      price=data.close[0]*0.5)
                print(self.order)
            else:
                print('\nSELLING')
                self.order = self.sell(data, exectype=bt.Order.Limit, size=0.5,
                                       price=data.close[0]*1.5)
                print(self.order)
    
    
        def notify_order(self, order):
            print('*' * 5, "NOTIFY ORDER", order)
    
    if __name__ == '__main__':
        cerebro = bt.Cerebro()
    
        exchange = 'bittrex'
        symbol = 'NEO/BTC'
    
        # Create broker
        broker_config = {'apiKey': 'xxxxxxxxxx',
                         'secret': 'xxxxxxxxxx',
                         'nonce': lambda: str(int(time.time() * 1000))
                        }
        broker = bt.brokers.CCXTBroker(exchange='bittrex',
                                       currency='BTC',
                                       config=broker_config)
        cerebro.setbroker(broker)
    
        # Create data feed
        hist_start_date = datetime.utcnow() - timedelta(minutes=30)
        data = bt.feeds.CCXT(exchange=exchange,
                             symbol=symbol,
                             timeframe=bt.TimeFrame.Minutes,
                             fromdate=hist_start_date,
                             compression=1)
    
        cerebro.adddata(data)
        cerebro.addstrategy(TestStrategy)
        cerebro.run()
    

    This successfully places an order on the exchange. However, the script crashes immediately afterward producing the following output:

    START
    prenext len 1 - counter 1
    prenext len 2 - counter 2
    prenext len 3 - counter 3
    prenext len 4 - counter 4
    prenext len 5 - counter 5
    ------ next len 6 - counter 5
    45.64666519054272 24.016532585430657 32.95714573240973
    ***** NEXT: 2018-09-03 15:40:00  0.00310706 0.00310706 0.00310706 0.00310706 0.5496506 Minute 6
    
    BUYING
    Traceback (most recent call last):
      File "C:/Users/pns21/Documents/Python Files/Beginner and Tutuorial files/Backtrader - Bartosh/Peter Test 2018.09.03.py", line 83, in <module>
        cerebro.run()
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run
        runstrat = self.runstrategies(iterstrat)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
        self._runnext(runstrats)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext
        strat._next()
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\strategy.py", line 325, in _next
        super(Strategy, self)._next()
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 268, in _next
        self.nextstart()  # only called for the 1st value
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 342, in nextstart
        self.next()
      File "C:/Users/pns21/Documents/Python Files/Beginner and Tutuorial files/Backtrader - Bartosh/Peter Test 2018.09.03.py", line 45, in next
        price=data.close[0]*0.5)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\strategy.py", line 917, in buy
        **kwargs)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\brokers\ccxtbroker.py", line 86, in buy
        return self._submit(owner, data, exectype, 'buy', size, price, kwargs)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\brokers\ccxtbroker.py", line 78, in _submit
        order = CCXTOrder(owner, data, _order)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\metabase.py", line 88, in __call__
        _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\metabase.py", line 78, in doinit
        _obj.__init__(*args, **kwargs)
      File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\stores\ccxtstore.py", line 39, in __init__
        self.size = float(ccxt_order['amount'])
    TypeError: float() argument must be a string or a number, not 'NoneType'
    

    Thanks in advance for any help solving this issue!!



  • @petens said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    size=0.5

    Does it work with size=1 ?



  • @ed-bartosh sadly it has the same result. The minimum order size on the bittrex exchange is 50,000 Satoshi or 0.0005 BTC, and either amount is over that for this symbol. Regardless the limit order is being successfully placed on the exchange. My inkling is that processing of the return values is causing this error, but i'm not experienced enough of a programmer to tell.



  • @Ed-Bartosh and others, I did some digging and I think I found the cause: the Bittrex API doesn't return any amount info.
    In the process of making an order ccxtstore.py calls the ccxt method "create_order" (at line 119 in the current version) :

    order = self.exchange.create_order(symbol=symbol, type=order_type, side=side,
                                               amount=amount, price=price, params=params)
    

    This works and places an order on the exchange, and returns a dictionary to "order" like this:

    print('order:', order)
    
    order: {'info': {'success': True, 'message': '', 'result': {'uuid': 'd9765e24-5d48-47b2-a343-4d487e679040'}}, 'id': 'd9765e24-5d48-47b2-a343-4d487e679040', 'symbol': 'NEO/BTC', 'type': 'limit', 'side': 'buy', 'status': 'open'}
    

    This is parsed with the ccxt method "parse_order" in the next command and returns a dict:

    parsed_order = self.exchange.parse_order(order['info'])
    print('parsed_order:', parsed_order)
    
    parsed_order: {'info': {'success': True, 'message': '', 'result': {'uuid': 'd9765e24-5d48-47b2-a343-4d487e679040'}}, 'id': None, 'timestamp': None, 'datetime': None, 'lastTradeTimestamp': None, 'symbol': None, 'type': 'limit', 'side': None, 'price': None, 'cost': None, 'average': None, 'amount': None, 'filled': None, 'remaining': None, 'status': None, 'fee': None}
    

    It's obvious bittrex doesn't return much info about the order. This dict is passed to "ccxt_order" on the call of the CCXTOrder class, and the script crashes while running:

            self.size = float(ccxt_order['amount'])
    

    from ccxtstore.py, because of the None value for the 'amount' key.
    Bittrex doesn't give the necessary info but there's other ways to get it with their API. Any ideas on how to fix this? Should this be handled in the ccxt method "parse_order"?

    On another note, what other crypto exchanges work well for people using this fork of backtrader?



  • @totoro

    Regarding the partial candle data, even though the PR wasn't merged, you can pick up fork of the Ed's and the others awesome work here:

    https://github.com/Dave-Vallance/bt-ccxt-store

    The repo contains made a few changes I made including an attempt to address the partial candle issue (although the solution is different to the PR). These changes work for me on the few exchanges I am looking at but your mileage may vary and I don't really plan on supporting it :)

    There are some other changes such as speeding up backfilling and allowing you to map some broker keys and strings if the defaults don't work amongst other things. (for example, since CCXT does not have a unified stop/limit order implementation, Kraken expects you to create a stoploss with 'stop-loss', whereas bitmex expects just 'stop'.

    For a full list of changes you can checkout the readme.



  • @petens Can you check if this commit fixes the issue for you?



  • Hi,

    I am trying to understand backtrader/ccxt/bitmex live behaviours.

    I am running a test script : simple strat with bitmex data feed in 1 min bar and bitmex broker.
    It computes a sma(period=2), waiting for this sma to be calculated and then send one single limit order.

    There is a delay I don't undertand:
    If I load 4 bars from bitmex, without setting a broker, historical data are loading quickly and strategy run quickly.
    With a broker it takes 1 min to run the strategy on 4 bars.
    If I need few days of backfill, it will take forever. Is it normal ?

    @ThatBlokeDave : Is that the kind of problem you add and corrected when you say " speeding up backfilling" ?

    Ouput without broker : = 5 sec to init

    time is  2018-09-10 10:54:33.984922  ## start
    init pausetrade =  True
    ***** DATA NOTIF: DELAYED
    time is  2018-09-10 10:54:38.298118  ## next
    ***** NEXT: 2018-09-10 08:51:00  6295.0 6295.0 6292.0 6292.5 365509.0 Minute 1
    time is  2018-09-10 10:54:38.299116  ## next
    ***** NEXT: 2018-09-10 08:52:00  6292.5 6292.5 6290.5 6290.5 496485.0 Minute 2
    time is  2018-09-10 10:54:38.300116  ## next
    ***** NEXT: 2018-09-10 08:53:00  6290.5 6290.5 6282.0 6282.0 2219366.0 Minute 3
    time is  2018-09-10 10:54:38.300116  ## next
    ***** NEXT: 2018-09-10 08:54:00  6282.0 6284.0 6282.0 6284.0 2435976.0 Minute 4
    ***** DATA NOTIF: LIVE
    time is  2018-09-10 10:55:03.080312  ## next
    ***** NEXT: 2018-09-10 08:55:00  6282.0 6282.0 6282.0 6282.0 0.0 Minute 5
    time is  2018-09-10 10:55:03.080312  ## SEND BUY LIMIT: 1 @ 5283.00
    Shutdown requested...exiting
    

    With a broker : = 44 sec to init !!

    time is  2018-09-10 12:13:18.374900  ## start
    init pausetrade =  True
    ***** DATA NOTIF: DELAYED
    time is  2018-09-10 12:13:31.147005  ## next
    ***** NEXT: 2018-09-10 10:10:00  6279.0 6279.0 6278.5 6279.0 546301.0 Minute 1
    time is  2018-09-10 12:13:41.502875  ## next
    ***** NEXT: 2018-09-10 10:11:00  6279.0 6279.5 6278.5 6279.5 335856.0 Minute 2
    time is  2018-09-10 12:13:51.868567  ## next
    ***** NEXT: 2018-09-10 10:12:00  6279.5 6279.5 6279.0 6279.0 141525.0 Minute 3
    time is  2018-09-10 12:14:02.329758  ## next
    ***** NEXT: 2018-09-10 10:13:00  6279.0 6279.5 6279.0 6279.0 21171.0 Minute 4
    ***** DATA NOTIF: LIVE
    time is  2018-09-10 12:14:16.795837  ## next
    ***** NEXT: 2018-09-10 10:14:00  6279.5 6279.5 6279.0 6279.0 20132.0 Minute 5
    time is  2018-09-10 12:14:20.980734  ## SEND BUY LIMIT: 1.0 @ 5279.00
    ***** NOTIFY ORDER 0
    time is  2018-09-10 12:15:08.587889  ## next
    ***** NEXT: 2018-09-10 10:15:00  6279.0 6279.0 6279.0 6279.0 0.0 Minute 6
    Shutdown requested...exiting
    

    Here is my code for this test :

    from datetime import datetime, timedelta
    import ccxt
    import backtrader as bt
    import os
    from pandas import bdate_range
    #get CCXT LOGS
    #import logging
    #logging.basicConfig(level=logging.DEBUG)
    
    class mmTest(bt.Strategy):
        params = (('pfast', 5), ('pslow', 30),)
        order = None
        pausetrade = True
    
        def log(self, txt, dt=None):
            ''' Logging function for this strategy'''
            dt = dt or self.datas[0].datetime.datetime(0)
            print('%s, %s' % (str(dt), txt))
    
        def __init__(self):
            # Keep a reference to the "close" line in the data[0] dataseries
            self.pausetrade = not self.data.haslivedata()
            print( "init pausetrade = " , self.pausetrade)
            self.dataclose = self.datas[0].close
            self.mm = bt.ind.MovingAverageSimple(self.data.close,period=2)
            self.order = None
    
        def notify_order(self, order):
            print('*' * 5, "NOTIFY ORDER", order.status )
            
        def notify_data(self, data, status, *args, **kwargs):
           self.pausetrade = (status != data.LIVE)
           print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args )
                   
        def sendBuyOrder(self):
            if self.pausetrade:
                return
            if self.mm[0] >0:
                limit=self.mm[0] - 1000
                ticksize = 0.5
                limit=round(limit/ticksize)*ticksize
                self.order =self.buy(price= limit, exectype=bt.Order.Limit,transmit=True)
                print('time is ',datetime.now(),' ## SEND BUY LIMIT: {:s} @ {:.2f}'.format(str(self.order.size),limit ))
            else:
                print('MM = %f' % (self.mm[0]))
     
        def next(self):
            print('time is ',datetime.now(),' ## next')
            for data in self.datas:
                print('*' * 5, 'NEXT:', bt.num2date(data.datetime[0]), data._name, data.open[0], data.high[0],data.low[0], data.close[0], data.volume[0], bt.TimeFrame.getname(data._timeframe), len(data))
            if not self.order :
                self.sendBuyOrder()
            
    try:
        print('time is ',datetime.now(),' ## start')
        
        cerebro = bt.Cerebro()
        
        feedconf = {'apiKey': '8KoTAFQZ3G1zkgRPSjj-LZi3', 'secret': ''}
        
        data = bt.feeds.CCXT(exchange="bitmex", symbol="BTC/USD",fromdate=datetime.now()+timedelta(hours = -2,minutes=-4), timeframe=bt.TimeFrame.Minutes,compression=1,config=feedconf,backfill_start=True)
    
        broker_config = {'apiKey': '8KoTAFQZ3G1zkgRPSjj-LZi3',
                             'secret': '****',}
        
        broker = bt.brokers.CCXTBroker(exchange='bitmex',
                                           currency='XBt',
                                           config=broker_config)
        cerebro.setbroker(broker)
        
        cerebro.adddata(data)
        cerebro.addstrategy(mmTest)
        cerebro.run()
        
        #cerebro.plot()
    
    except KeyboardInterrupt:
        print("Shutdown requested...exiting")
    #except Exception:
    #    traceback.print_exc(file=sys.stdout)
    quit
    
    


  • Hey @ed-bartosh, that does seem to do the trick, a million thanks! I'll keep looking for issues and fixes to contribute as I go along using this.



  • @søren-pallesen
    Hi,
    Did you get correct speed for backfilling few minutes bars when you use Bitmex feed AND Bitmex broker?
    It is seems that 1-minute bars are received in one block but a simple strategy with 1 indicator needs at least 10sec per bar to run/compute the "next" function.

    May be, it is the normal behaviour of Backtrader running live?



  • Hi, not sure if this is the right place to ask but how do you input backfeed data from ccxt into backtrader? Do I have to convert the ccxt output to CSV and proceed from there?

    import ccxt
    
    bitmex = ccxt.bitmex()
    
    # params:
    symbol = 'BTC/USD'
    timeframe = '1d'
    limit = 750
    params = {'partial': False}  # ←--------  no reversal
    since = None
    
    candles = bitmex.fetch_ohlcv(symbol, timeframe, since, limit, params)
    

    Here I want to use the data in candles as feed for backtesting (not live trading). Thank you!



  • @thomaslee
    You can probably convert candles into a Pandas dataframe with datetime, open, high, low , close, volume.
    And then use this feed :

    data0=bt.feeds.PandasDirectData(dataname=dataframe,**kwargs,datetime=0,time=1,open=2,high=3,low=4,close=5,volume=6,openinterest=-1,timeframe=bt.TimeFrame.Days,compression=1)
    


  • @sirhc

    Is that the kind of problem you add and corrected when you say " speeding up backfilling" ?

    Yes - That is correct. The reason it is slow is because the broker will try to check your cash/balance on every call of next(). (If I remember correctly it makes 2 or 3 checks in total between each call of next). Since, the store will make use rest API calls to do these checks, this can add quite a few seconds delay between each call of next(). (Because of the rest API rate limit).

    So in my fork, I cheat a bit. The cash and balance checks return the last values received from the last rest call. They do not make another call. Instead, the user must manually check the balance if they need up to date balance information before entering a position. (at which point the latest values will be saved).

    Not an ideal solution but it works for me.



  • @ed-bartosh said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    guide

    Do you ever experience the problem of self.order = self.buy(size=1.5) or in the buy method ,the param size can not set to a float value ?


Log in to reply
 

});