For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Anyone use backtrader to do live trading on Bitcoin exchange?



  • @ed-bartosh said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    Hi,

    Latest project update:

    Please, install it from ccxt branch of my github repo:

    pip install pip install --upgrade git+https://github.com/bartosh/backtrader.git@ccxt
    

    NOTE: due to the change in ccxt API please upgrade your ccxt installation to the latest ccxt master using this command:

    pip install ccxt
    

    Good luck :) !

    Ed

    Hi Ed,

    Thank you for the update! I am running into the following error after the update:

    START
    prenext len 1 - counter 1
    prenext len 2 - counter 2
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 347, in fetch
        response.raise_for_status()
      File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 935, in raise_for_status
        raise HTTPError(http_error_msg, response=self)
    requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://api.bitfinex.com/v1/balances
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "machine_engin_prod.py", line 167, in <module>
        cerebro.run()
      File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1127, in run
        runstrat = self.runstrategies(iterstrat)
      File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1295, in runstrategies
        self._runnext(runstrats)
      File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1626, in _runnext
        strat._next()
      File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 329, in _next
        self._next_observers(minperstatus)
      File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 357, in _next_observers
        observer._next()
      File "/usr/local/lib/python3.6/dist-packages/backtrader/lineiterator.py", line 275, in _next
        self.next()
      File "/usr/local/lib/python3.6/dist-packages/backtrader/observers/broker.py", line 111, in next
        self.lines.cash[0] = self._owner.broker.getcash()
      File "/usr/local/lib/python3.6/dist-packages/backtrader/brokers/ccxtbroker.py", line 60, in getcash
        return self.store.getcash(self.currency)
      File "/usr/local/lib/python3.6/dist-packages/backtrader/stores/ccxtstore.py", line 87, in retry_method
        return method(self, *args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/backtrader/stores/ccxtstore.py", line 96, in getcash
        return self.exchange.fetch_balance()['free'][currency]
      File "/usr/local/lib/python3.6/dist-packages/ccxt/bitfinex.py", line 354, in fetch_balance
        balances = self.privatePostBalances()
      File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 297, in request
        return self.fetch2(path, api, method, params, headers, body)
      File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 294, in fetch2
        return self.fetch(request['url'], request['method'], request['headers'], request['body'])
      File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 359, in fetch
        self.handle_errors(response.status_code, response.reason, url, method, None, self.last_http_response)
      File "/usr/local/lib/python3.6/dist-packages/ccxt/bitfinex.py", line 725, in handle_errors
        exact = self.exceptions.exact
    AttributeError: 'dict' object has no attribute 'exact'
    

    Also I believe the correct pip commands are;

    pip install --upgrade git+https://github.com/bartosh/backtrader.git@ccxt
    

    and

    pip install --upgrade ccxt
    

    Thank you in advance!



  • @kazi can you show your test script?



  • @ed-bartosh

    I just did a bit of further investigation and appears the current rateLimit set in backtrader/stores/ccxtstore.py will trigger bitfinex's DDOS protection. From what I have read (https://bitcoin.stackexchange.com/questions/36952/bitfinex-api-limit) they allow 60 requests per minute. here is a script that should trigger it:

    # -*- coding: utf-8; py-indent-offset:4 -*-
    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    import time
    from datetime import datetime, timedelta
    import backtrader as bt
    
    
    class TestStrategy(bt.Strategy):
        def start(self):
            self.counter = 0
            print('START')
    
        def prenext(self):
            self.counter += 1
            print('prenext len %d - counter %d' % (len(self), self.counter))
    
        def __init__(self):
            # Keep a reference to the "close" line in the data[0] dataseries
            self.dataclose = self.data.close
            self.datahigh = self.data.high
            self.datalow = self.data.low
            self.datavolume = self.data.volume
    
        def next(self):
            print('------ next len %d - counter %d' % (len(self), self.counter))
    
            self.counter += 1
                
            print('*' * 5, 'NEXT:', bt.num2date(self.data0.datetime[0]), 'Symbol: ',self.data0._name, 'Open: ', self.data0.open[0], 'High: ', self.data0.high[0],
                  'Low: ', self.data0.low[0], 'Close: ', self.data0.close[0], 'Volume: ', self.data0.volume[0],
                  bt.TimeFrame.getname(self.data0._timeframe), len(self.data0))
    
    if __name__ == '__main__':
        cerebro = bt.Cerebro()
        
        hist_start_date = datetime.utcnow() - timedelta(minutes=60)
        data_ticks = bt.feeds.CCXT(exchange='bitfinex', symbol='EOS/BTC', name="eos_btc_tick",
                               timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date, compression=5)
    
        cerebro.adddata(data_ticks, name="eos_btc_min")
    
        cerebro.addstrategy(TestStrategy)
        cerebro.run()
    

    Thanks in advance



  • Thx @Ed-Bartosh really useful update !



  • @kazi said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    @ed-bartosh

    I just did a bit of further investigation and appears the current rateLimit set in backtrader/stores/ccxtstore.py will trigger bitfinex's DDOS protection. From what I have read (https://bitcoin.stackexchange.com/questions/36952/bitfinex-api-limit) they allow 60 requests per minute.

    ccxt store uses default ccxt rate limit, so it's better to report it to ccxt folks. They're using rate limit 1500 as far as I can see here: https://github.com/ccxt/ccxt/blob/master/python/ccxt/bitmex.py on line 21.

    As a temporary workaround you can use ohlcv_limit feed parameter to change the limit.



  • @ed-bartosh said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    As a temporary workaround you can use ohlcv_limit feed parameter to change the limit.

    Sorry, ohlcv_limit is another thing. You can use config={"rateLimit": <ratelimit value>} to overwrite default exchange settings.

    Your script worked for me with rateLimit 2000:

    data_ticks = bt.feeds.CCXT(exchange='bitfinex',
                               symbol='EOS/BTC',
                               name="eos_btc_tick",
                               timeframe=bt.TimeFrame.Minutes, 
                               fromdate=hist_start_date,
                               compression=5,
                               config={'rateLimit': 2000})
    


  • @ed-bartosh said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    Sorry, ohlcv_limit is another thing. You can use config={"rateLimit": <ratelimit value>} to overwrite default exchange settings.

    Your script worked for me with rateLimit 2000:

    data_ticks = bt.feeds.CCXT(exchange='bitfinex',
                               symbol='EOS/BTC',
                               name="eos_btc_tick",
                               timeframe=bt.TimeFrame.Minutes, 
                               fromdate=hist_start_date,
                               compression=5,
                               config={'rateLimit': 2000})
    

    Thanks the following did the trick for me:

    config={'rateLimit': 10000, 'enableRateLimit': True}
    

    Will start testing tomorrow. Will let you know if I come across anything.

    Thanks again!



  • Since Yesteday I am getting an error when getting klines (candles) from binance (used for live trading).
    The following request raises the following error from binance api:
    https://api.binance.com/api/v1/klines?symbol=AEBTC&interval=1d&limit=None&startTime=512397977964
    {"code":-1100,"msg":"Illegal characters found in parameter 'limit'; legal range is '^[0-9]{1,20}$'."}

    Removing "limit=None&' solves the problem, as well as setting limit to an integer.

    I have fixed the problem by inserting the following code in ccxt.py after line 61:

        if ohlcv_limit == None and exchange == 'binance':
            ohlcv_limit = 99999


  • @benoît-zuber Thanks for the error report. You don't need to change feed code to change olcv_limit. There is a feed parameter 'ohlcv_limit' exactly for this purpose.



  • @ed-bartosh Great, thanks. I didn't realise it before. Problem solved.



  • Project update:

    • ccxt branch has been rebased on top of the latest backtrader tag 1.9.61.122
    • implemented retrying on exchange errors

    Please upgrade your installations from ccxt branch of my github repository:

    pip install --upgrade git+https://github.com/bartosh/backtrader.git@ccxt
    


  • @ed-bartosh Hi Ed / All. Just jumping into this discussion after installing backtrader with the aim of backtesting and tarding using binance, bittrex or poloniex. I have installed ccxt as suggested above - where can I find a sample script that uses one of these exchanges as input and runs through a basic strategy? Thanks in advance!



  • Hi,
    I see many people having this problem but I can't see a fix for it.

    After installing and even trying to upgrade with pip install --upgrade git+https://github.com/bartosh/backtrader.git@ccxt

    I still can't import ccxtbroker

    Running:

    python -c 'from backtrader.brokers.ccxtbroker import CCXTBroker'
    

    I get:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/brokers/ccxtbroker.py", line 26, in <module>
        from backtrader.stores.ccxtstore import CCXTStore
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py", line 27, in <module>
        import ccxt
    ModuleNotFoundError: No module named 'ccxt'
    

    I tried to reinstall everything, both using pip and using setup.py, but the result is always the same.



  • @bruno-morgado Ok, I found the issue. I didn't install the CCXT package beforehand. Perhaps introduce that as a requirement for backtester@ccxt installation so that in the future new people don't have that issue?



  • I can't use Binance because when I try to check out the present open orders with: print('ORDERS:', self.broker.get_orders_open()), I get the error:

    Traceback (most recent call last):
      File "ccxtbot/main.py", line 101, in <module>
        sys.exit(runstrategy(sys.argv))
      File "ccxtbot/main.py", line 95, in runstrategy
        cerebro.run()
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/cerebro.py", line 1127, in run
        runstrat = self.runstrategies(iterstrat)
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/cerebro.py", line 1214, in runstrategies
        strat = stratcls(*sargs, **skwargs)
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/metabase.py", line 88, in __call__
        _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/metabase.py", line 78, in doinit
        _obj.__init__(*args, **kwargs)
      File "ccxtbot/main.py", line 21, in __init__
        print('ORDERS:', self.broker.get_orders_open())
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/brokers/ccxtbroker.py", line 102, in get_orders_open
        return self.store.fetch_open_orders()
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py", line 87, in retry_method
        return method(self, *args, **kwargs)
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py", line 125, in fetch_open_orders
        return self.exchange.fetchOpenOrders()
      File "/home/morgado/.conda/envs/ccxtbot/lib/python3.6/site-packages/ccxt/binance.py", line 709, in fetch_open_orders
        raise ExchangeError(self.id + ' fetchOpenOrders WARNING: fetching open orders without specifying a symbol is rate-limited to one call per ' + str(fetchOpenOrdersRateLimit) + ' seconds. Do not call self method frequently to avoid ban. Set ' + self.id + '.options["warnOnFetchOpenOrdersWithoutSymbol"] = False to suppress self warning message.')
    ccxt.base.errors.ExchangeError: binance fetchOpenOrders WARNING: fetching open orders without specifying a symbol is rate-limited to one call per 136 seconds. Do not call self method frequently to avoid ban. Set binance.options["warnOnFetchOpenOrdersWithoutSymbol"] = False to suppress self warning message.
    

    Any ideas on how to solve this?



  • @bruno-morgado Can you show simple script to reproduce this?



  • @bruno-morgado

    I found the issue. I didn't install the CCXT package beforehand.

    I posted installation instructions here some time ago. They inlude ccxt installation as far as I remember.

    Perhaps introduce that as a requirement for backtester@ccxt installation

    Unfortunately I can't do that in the plugin. Modifying backtrader's setup.py is not good either as I'm trying to avoid modifying backtrader code as much as I can.



  • Hello All,
    I am just having a play around with this fork. (Great job btw @Ed-Bartosh)

    Is anyone seeing sync issues after a script has been running 24+ hours?

    I was running a test and noticed that I lost sync by about 2 hours overnight. My first guess is that it might have something to do with fetching data at regular rate limited intervals rather than streaming. I could be well wide of the mark though with that comment.

    Here is my output: I terminated and restarted the script within a few seconds of each other. You can see the time is 03:17 when I terminated and05:00` when I restarted it.

    ---------------------------- NEXT ----------------------------------
    1: Data Name:                            btc_usd_min
    2: Bar Num:                              1087
    3: Current date:                         2018-03-09 03:16:00
    4: Open:                                 8990.1
    5: High:                                 8990.1
    6: Low:                                  8973.0
    7: Close:                                8978.5
    8: Volume:                               20.92727129
    --------------------------------------------------------------------
    ---------------------------- NEXT ----------------------------------
    1: Data Name:                            btc_usd_min
    2: Bar Num:                              1088
    3: Current date:                         2018-03-09 03:17:00
    4: Open:                                 8977.5
    5: High:                                 8987.4
    6: Low:                                  8973.0
    7: Close:                                8973.6
    8: Volume:                               10.42514496
    --------------------------------------------------------------------
    Traceback (most recent call last):
      File ".\streaming.py", line 40, in <module>
        sys.exit(runstrategy(sys.argv))
      File ".\streaming.py", line 37, in runstrategy
        cerebro.run()
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        runstrat = self.runstrategies(iterstrat)
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        self._runnext(runstrats)
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        drets.append(d.next(ticks=False))
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        ret = self.load()
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        _loadret = self._load()
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        self._fetch_ohlcv()
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        since=since, limit=limit)):
      File "C:\Users\Dave\Google Drive\Development\Python\Python3\personal-
        time.sleep(self.exchange.rateLimit / 1000)
    KeyboardInterrupt
    PS C:\Users\Dave\Google Drive\Development\Python\Python3\personal-algos
    ---------------------------- NEXT ----------------------------------
    1: Data Name:                            btc_usd_min
    2: Bar Num:                              1
    3: Current date:                         2018-03-09 05:00:00
    4: Open:                                 8639.0
    5: High:                                 8639.0
    6: Low:                                  8583.91298351
    7: Close:                                8595.5
    8: Volume:                               113.38661117
    --------------------------------------------------------------------
    ---------------------------- NEXT ----------------------------------
    1: Data Name:                            btc_usd_min
    2: Bar Num:                              2
    3: Current date:                         2018-03-09 05:01:00
    4: Open:                                 8595.2
    5: High:                                 8595.2
    6: Low:                                  8560.0
    7: Close:                                8587.0
    8: Volume:                               220.58504529
    --------------------------------------------------------------------
    

    Here is the code used to produce it.

    import sys
    
    from datetime import datetime, timedelta
    import backtrader as bt
    
    class TestStrategy(bt.Strategy):
        def next(self):
            for data in self.datas:
                print('---------------------------- NEXT ----------------------------------')
                print("1: Data Name:                            {}".format(data._name))
                print("2: Bar Num:                              {}".format(len(data)))
                print("3: Current date:                         {}".format(data.datetime.datetime()))
                print('4: Open:                                 {}'.format(data.open[0]))
                print('5: High:                                 {}'.format(data.high[0]))
                print('6: Low:                                  {}'.format(data.low[0]))
                print('7: Close:                                {}'.format(data.close[0]))
                print('8: Volume:                               {}'.format(data.volume[0]))
                print('--------------------------------------------------------------------')
    
    
    # Create a cerebro
    cerebro = bt.Cerebro()
    
    # Create data feeds
    hist_start_date = datetime.utcnow() - timedelta(minutes=3)
    data_min = bt.feeds.CCXT(exchange="bitfinex2", symbol="BTC/USD", name="btc_usd_min",
                             timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date,
                             config={'rateLimit': 10000, 'enableRateLimit': True}) #, historical=True)
    cerebro.adddata(data_min)
    # Add the strategy
    cerebro.addstrategy(TestStrategy)
    
    # Run the strategy
    cerebro.run()
    
    


  • Hi everybody,

    @Ed-Bartosh great work.

    There seems to be a problem when fetching live data. The backfilling works correctly, but once the live data is used the feeds receives the data of the current minute (assuming we are trading minute data) that is not yet finished. Essentially the opening value is correct, but the rest (close high low vol) is not.

    With the following code:

    # !/usr/bin/env python
    # -*- coding: utf-8; py-indent-offset:4 -*-
    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    import sys
    import time
    from datetime import datetime, timedelta
    
    import backtrader as bt
    import ccxt
    
    #pylint: disable=E1101,E1123
    
    class TestStrategy(bt.Strategy):
    
        def start(self):
            self.counter = 0
            print('START')
    
        def prenext(self):
            self.counter += 1
            print('prenext len %d - counter %d' % (len(self), self.counter))
    
        def __init__(self):
            pass
    
        def next(self):
            print('------ next len %d - counter %d' % (len(self), self.counter))
    
            self.counter += 1
    
            print('*' * 5, 'NEXT:', bt.num2date(self.data0.datetime[0]),
                  self.data0._name, self.data0.open[0], self.data0.high[0],
                  self.data0.low[0], self.data0.close[0], self.data0.volume[0],
                  bt.TimeFrame.getname(self.data0._timeframe), len(self.data0))
    
    if __name__ == '__main__':
        cerebro = bt.Cerebro()
    
        exchange = sys.argv[1] if len(sys.argv) > 1 else 'binance'
        symbol = sys.argv[2] if len(sys.argv) > 2 else 'BTC/USDT'
    
        hist_start_date = datetime.utcnow() - timedelta(minutes=3)
        data = bt.feeds.CCXT(exchange=exchange,
                             symbol=symbol,
                             timeframe=bt.TimeFrame.Minutes,
                             fromdate=hist_start_date,
                             ohlcv_limit=999)
    
        cerebro.adddata(data)
        cerebro.addstrategy(TestStrategy)
        cerebro.run()
    

    I get the following output:

    START
    ------ next len 1 - counter 0
    ***** NEXT: 2018-03-09 14:40:00  9090.0 9115.35 9080.0 9114.99 30.29132 Minute 1
    ------ next len 2 - counter 1
    ***** NEXT: 2018-03-09 14:41:00  9112.25 9148.0 9090.11 9102.02 73.48228 Minute 2
    ------ next len 3 - counter 2
    ***** NEXT: 2018-03-09 14:42:00  9109.99 9110.0 9097.0 9097.0 35.020203 Minute 3
    ------ next len 4 - counter 3
    ***** NEXT: 2018-03-09 14:43:00  9099.0 9100.0 9090.43 9090.43 2.0025 Minute 4
    ------ next len 5 - counter 4
    ***** NEXT: 2018-03-09 14:44:00  9099.0 9099.0 9095.0 9095.0 0.058638 Minute 5
    ------ next len 6 - counter 5
    ***** NEXT: 2018-03-09 14:45:00  9088.78 9088.78 9088.78 9088.78 0.135 Minute 6
    ------ next len 7 - counter 6
    ***** NEXT: 2018-03-09 14:46:00  9099.0 9100.9 9099.0 9100.9 0.0327 Minute 7
    

    The first 2 minutes are correct, because they have completely finished. Afterwards the values are off because the minute data is obtained when the minute is still running. This is essentially some 1off error. If you want to check for yourself binance - BTC/USDT

    I also checked for bitfinex (python script.py bitfinex2 BTC/USD). Same issue.

    I don't know if this is related to the ccxt/backtrader branch or related to ccxt master. @Ed-Bartosh let me know if you want me to look into it.



  • @fivo said in Anyone use backtrader to do live trading on Bitcoin exchange?:

    There seems to be a problem when fetching live data. The backfilling works correctly, but once the live data is used the feeds receives the data of the current minute (assuming we are trading minute data) that is not yet finished. Essentially the opening value is correct, but the rest (close high low vol) is not.

    It doesn't seem to be ccxt as the following little script produces the correct values. I think the code around this line seems to be fetching data as soon as some data of the currently running minute is available. I will try look into it.

    # !/usr/bin/env python
    # -*- coding: utf-8; py-indent-offset:4 -*-
    
    from datetime import datetime, timedelta
    import time
    
    import ccxt
    
    binance = ccxt.binance()
    
    while True:
        fromdate = datetime.utcnow() - timedelta(minutes=2)
        since = int((fromdate - datetime(1970, 1, 1)).total_seconds() * 1000)
        ohlcvs = binance.fetch_ohlcv('BTC/USDT', '1m', since=since, limit=1)
        if not ohlcvs:
            continue
    
        ohlcv = ohlcvs[0]
    
        print(datetime.utcfromtimestamp(ohlcv[0]/1000), ohlcv[1], ohlcv[2],
              ohlcv[3], ohlcv[4], ohlcv[5])
    
        time.sleep(1)
    

    Can anybody reproduce this issue?


Log in to reply
 

});