Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    dealing with ExchangeError in backtrader ccxt branch

    General Discussion
    3
    11
    3570
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      Benoît Zuber last edited by Benoît Zuber

      @Ed-Bartosh Hi, congrats for this branch, it is fantastic! I was just just wondering how you deal with ExchangeError that occur randomly from time to time. Namely bittrex api sometimes returns an empty reply and cerebro.run() crashes. Is there a trick to catch the exception and ignore it without touching the source code of backtrader or ccxt?
      Here is my dummy example:

      class TestStrategy(bt.Strategy):
          def next(self):
              print("time now: %s." %time.strftime("%H:%M:%S", time.gmtime()))
              for data in self.datas:
                  print(bt.num2date(data.datetime[0]), data._name, data.open[0], data.high[0],
                        data.low[0], data.close[0], data.volume[0],
                        "day", len(data))
      def runstrategy():
          # Create a cerebro
          cerebro = bt.Cerebro()
          
          # Create a broker
          hist_start_date = datetime.utcnow() - timedelta(days=10)
          #broker = bt.brokers.CCXTBroker(exchange='bittrex')
          data_min1 = bt.feeds.CCXT(exchange = 'bittrex', symbol='BTC/USDT', 
                                    name="btc_usdt_bittrex", timeframe=bt.TimeFrame.Days,
                                    compression = 1,
                                    fromdate = hist_start_date)
          data_min2 = bt.feeds.CCXT(exchange = 'gdax', symbol='BTC/USD', 
                                    name="btc_usdt_gdax", timeframe=bt.TimeFrame.Days,
                                    compression = 1,
                                    fromdate = hist_start_date)
          cerebro.adddata(data_min1)
          cerebro.adddata(data_min2)
          cerebro.addstrategy(TestStrategy)
          cerebro.run()
      

      When I run strategy() I am getting this output:

      time now: 21:33:20.
      2018-02-10 00:00:00 btc_usdt_bittrex 8689.0 9041.40405 8140.0 8540.0 6842.4350137 day 1
      2018-02-10 00:00:00 btc_usdt_gdax 8671.0 9090.0 8155.0 8547.49 6763.749508230131 day 1
      time now: 21:33:20.
      2018-02-11 00:00:00 btc_usdt_bittrex 8540.00000001 8551.0 7815.73000001 8052.5 4819.85510126 day 2
      2018-02-11 00:00:00 btc_usdt_gdax 8547.48 8547.49 7851.0 8072.99 5030.079040800108 day 2
      time now: 21:33:20.
      2018-02-12 00:00:00 btc_usdt_bittrex 8052.5 8980.0 8052.0 8913.48826283 4337.03012567 day 3
      2018-02-12 00:00:00 btc_usdt_gdax 8073.0 8950.0 8072.99 8872.28 5192.890664420066 day 3
      time now: 21:33:20.
      2018-02-13 00:00:00 btc_usdt_bittrex 8913.48826283 8945.0 8359.0 8512.00000005 3382.11671751 day 4
      2018-02-13 00:00:00 btc_usdt_gdax 8872.27 8925.0 8393.1 8520.01 4254.992621180045 day 4
      time now: 21:33:20.
      2018-02-14 00:00:00 btc_usdt_bittrex 8512.00000005 9487.6 8512.00000004 9450.0 4072.19201411 day 5
      2018-02-14 00:00:00 btc_usdt_gdax 8520.01 9482.5 8520.0 9472.98 6591.748869050115 day 5
      time now: 21:33:20.
      2018-02-15 00:00:00 btc_usdt_bittrex 9450.0 10190.00000002 9330.0 10035.00000004 5343.13567805 day 6
      2018-02-15 00:00:00 btc_usdt_gdax 9472.98 10249.81 9342.46 10031.23 7688.429391570176 day 6
      time now: 21:33:20.
      2018-02-16 00:00:00 btc_usdt_bittrex 10035.00000004 10287.0 9660.0 10152.0 3018.51905787 day 7
      2018-02-16 00:00:00 btc_usdt_gdax 10027.8 10307.68 9730.01 10167.49 5089.095234060047 day 7
      time now: 21:33:20.
      2018-02-17 00:00:00 btc_usdt_bittrex 10150.0 11098.0 10049.00000201 11048.4 4079.41257877 day 8
      2018-02-17 00:00:00 btc_usdt_gdax 10167.49 11149.0 10057.0 11121.5 5717.07199700005 day 8
      time now: 21:33:20.
      2018-02-18 00:00:00 btc_usdt_bittrex 11048.99999998 11245.0 10080.99999998 10380.00000009 5746.00081324 day 9
      2018-02-18 00:00:00 btc_usdt_gdax 11121.5 11299.1 10159.0 10380.04 6455.1921786601015 day 9
      time now: 21:33:20.
      2018-02-19 00:00:00 btc_usdt_bittrex 10380.00000009 11238.23760818 10303.45443897 11130.0 3365.1113783 day 10
      2018-02-19 00:00:00 btc_usdt_gdax 10380.04 11274.11 10297.39 11102.5 4042.6038281900164 day 10
      time now: 00:01:02.
      2018-02-19 00:00:00 btc_usdt_bittrex 10380.00000009 11238.23760818 10303.45443897 11130.0 3365.1113783 day 10
      2018-02-20 00:00:00 btc_usdt_gdax 11140.0 11140.0 11139.99 11140.0 1.67212084 day 11
      time now: 00:02:40.
      2018-02-20 00:00:00 btc_usdt_bittrex 11159.999 11160.0 11159.999 11159.999 0.6325051 day 11
      2018-02-20 00:00:00 btc_usdt_gdax 11140.0 11140.0 11139.99 11140.0 1.67212084 day 11
      
      ---------------------------------------------------------------------------
      ExchangeError                             Traceback (most recent call last)
      <ipython-input-5-8a8200ffe2f4> in <module>()
            1 #th = Thread(target = runstrategy)
            2 #th.start()
      ----> 3 runstrategy()
      
      <ipython-input-4-45bb8d84d254> in runstrategy()
           17     cerebro.adddata(data_min2)
           18     cerebro.addstrategy(TestStrategy)
      ---> 19     cerebro.run()
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/cerebro.py in run(self, **kwargs)
         1125             # let's skip process "spawning"
         1126             for iterstrat in iterstrats:
      -> 1127                 runstrat = self.runstrategies(iterstrat)
         1128                 self.runstrats.append(runstrat)
         1129         else:
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/cerebro.py in runstrategies(self, iterstrat, predata)
         1293                     self._runnext_old(runstrats)
         1294                 else:
      -> 1295                     self._runnext(runstrats)
         1296 
         1297             for strat in runstrats:
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/cerebro.py in _runnext(self, runstrats)
         1536                 qlapse = datetime.datetime.utcnow() - qstart
         1537                 d.do_qcheck(newqcheck, qlapse.total_seconds())
      -> 1538                 drets.append(d.next(ticks=False))
         1539 
         1540             d0ret = any((dret for dret in drets))
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/feed.py in next(self, datamaster, ticks)
          402 
          403             # not preloaded - request next bar
      --> 404             ret = self.load()
          405             if not ret:
          406                 # if load cannot produce bars - forward the result
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/feed.py in load(self)
          474 
          475             if not self._fromstack(stash=True):
      --> 476                 _loadret = self._load()
          477                 if not _loadret:  # no bar use force to make sure in exactbars
          478                     # the pointer is undone this covers especially (but not
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/feeds/ccxt.py in _load(self)
           89                     return self._load_ticks()
           90                 else:
      ---> 91                     self._fetch_ohlcv()
           92                     return self._load_ohlcv()
           93             elif self._state == self._ST_HISTORBACK:
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/feeds/ccxt.py in _fetch_ohlcv(self, fromdate)
          123             dlen = len(self._data)
          124             for ohlcv in sorted(self.store.fetch_ohlcv(self.symbol, timeframe=granularity,
      --> 125                                                        since=since, limit=limit)):
          126                 if None in ohlcv:
          127                     continue
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py in retry_method(self, *args, **kwargs)
           85                 time.sleep(self.exchange.rateLimit / 1000)
           86                 try:
      ---> 87                     return method(self, *args, **kwargs)
           88                 except NetworkError:
           89                     if i == self.retries - 1:
      
      ~/anaconda3/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py in fetch_ohlcv(self, symbol, timeframe, since, limit)
          119     @retry
          120     def fetch_ohlcv(self, symbol, timeframe, since, limit):
      --> 121         return self.exchange.fetch_ohlcv(symbol, timeframe=timeframe, since=since, limit=limit)
          122 
          123     @retry
      
      ~/anaconda3/lib/python3.6/site-packages/ccxt/bittrex.py in fetch_ohlcv(self, symbol, timeframe, since, limit, params)
          402             if response['result']:
          403                 return self.parse_ohlcvs(response['result'], market, timeframe, since, limit)
      --> 404         raise ExchangeError(self.id + ' returned an empty or unrecognized response: ' + self.json(response))
          405 
          406     def fetch_open_orders(self, symbol=None, since=None, limit=None, params={}):
      
      ExchangeError: bittrex returned an empty or unrecognized response: {"success":true,"message":"","result":null}
      
      E 1 Reply Last reply Reply Quote 0
      • E
        Ed Bartosh @Benoît Zuber last edited by

        @benoît-zuber I'm afraid ExchangeError can't be avoided in the strategy code. However, it might make sense to do in ccxt store. Can you try if this patch fixes the issue for you?

        diff --git a/backtrader/stores/ccxtstore.py b/backtrader/stores/ccxtstore.py
        index 16f6406..b58e0a0 100644
        --- a/backtrader/stores/ccxtstore.py
        +++ b/backtrader/stores/ccxtstore.py
        @@ -25,7 +25,7 @@ import time
         from functools import wraps
         
         import ccxt
        -from ccxt.base.errors import NetworkError
        +from ccxt.base.errors import NetworkError, ExchangeError
         
         import backtrader as bt
         
        @@ -85,7 +85,7 @@ class CCXTStore(object):
                         time.sleep(self.exchange.rateLimit / 1000)
                         try:
                             return method(self, *args, **kwargs)
        -                except NetworkError:
        +                except NetworkError, ExchangeError:
                             if i == self.retries - 1:
                                 raise
        
        
        B 1 Reply Last reply Reply Quote 0
        • B
          Benoît Zuber @Ed Bartosh last edited by

          @ed-bartosh Sure I am going to try. How can you specify the maximum number of retries? From my previous experience it needs to retry for about one minute in some cases.

          E 1 Reply Last reply Reply Quote 0
          • E
            Ed Bartosh @Benoît Zuber last edited by

            @benoît-zuber to specify number of retries you can use 'retries' feed parameter.

            1 Reply Last reply Reply Quote 0
            • S
              Søren Pallesen last edited by

              @Ed-Bartosh will this fix get merged into your main CCXT branch?

              E 1 Reply Last reply Reply Quote 0
              • E
                Ed Bartosh @Søren Pallesen last edited by

                @søren-pallesen Yes, I'm going to merge it after @Benoît-Zuber confirms that it works.

                1 Reply Last reply Reply Quote 1
                • B
                  Benoît Zuber last edited by

                  @Ed-Bartosh I have tested your patch and it works perfectly. I added a print statement at in the except block. I paste below a part of the output. Note that I set retries to 100 in the feeds.

                  2018-02-27, BUY PURA/BTC: size is 1852.797522, value is 0.148205.
                  2018-02-27, o:7.999e-05, h:8e-05, l:6.8e-05, c:6.82e-05
                  2018-02-27, previous highest close : 7.999e-05, atr7.407824056208675e-06
                  2018-02-27, OPERATION PROFIT for LTC/BTC, GROSS -0.03, NET -0.03, ACCOUNT VALUE 0.92622656
                  2018-02-27, prenext
                  2018-02-27, 1 open trade(s): PURA/BTC.
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 6; maximum retries: 100
                  retrying, i = 7; maximum retries: 100
                  retrying, i = 8; maximum retries: 100
                  retrying, i = 9; maximum retries: 100
                  retrying, i = 10; maximum retries: 100
                  retrying, i = 11; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 6; maximum retries: 100
                  retrying, i = 7; maximum retries: 100
                  retrying, i = 8; maximum retries: 100
                  retrying, i = 9; maximum retries: 100
                  retrying, i = 10; maximum retries: 100
                  retrying, i = 11; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 6; maximum retries: 100
                  retrying, i = 7; maximum retries: 100
                  retrying, i = 8; maximum retries: 100
                  retrying, i = 9; maximum retries: 100
                  retrying, i = 10; maximum retries: 100
                  retrying, i = 11; maximum retries: 100
                  retrying, i = 12; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 6; maximum retries: 100
                  retrying, i = 7; maximum retries: 100
                  retrying, i = 8; maximum retries: 100
                  retrying, i = 9; maximum retries: 100
                  retrying, i = 10; maximum retries: 100
                  retrying, i = 11; maximum retries: 100
                  retrying, i = 12; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 6; maximum retries: 100
                  retrying, i = 7; maximum retries: 100
                  retrying, i = 8; maximum retries: 100
                  retrying, i = 9; maximum retries: 100
                  retrying, i = 10; maximum retries: 100
                  retrying, i = 11; maximum retries: 100
                  retrying, i = 12; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 6; maximum retries: 100
                  retrying, i = 7; maximum retries: 100
                  retrying, i = 8; maximum retries: 100
                  retrying, i = 9; maximum retries: 100
                  retrying, i = 10; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 1; maximum retries: 100
                  retrying, i = 2; maximum retries: 100
                  retrying, i = 3; maximum retries: 100
                  retrying, i = 4; maximum retries: 100
                  retrying, i = 5; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  retrying, i = 0; maximum retries: 100
                  2018-02-27, prenext
                  2018-02-27, 1 open trade(s): PURA/BTC.
                  

                  I just had to put NetworkError and ExchangeError in a tuple at the except line. I am using python 3.6.

                  BTW It seems that backtrader keeps asking the exchanges for the latest daily candle. This seems like a waste of resources. Wouldn't it make sense that the program only contacts the exchanges at midnight?

                  E 1 Reply Last reply Reply Quote 0
                  • E
                    Ed Bartosh @Benoît Zuber last edited by

                    @benoît-zuber said in dealing with ExchangeError in backtrader ccxt branch:

                    Wouldn't it make sense that the program only contacts the exchanges at midnight?

                    Midnight in which time zone?

                    B 1 Reply Last reply Reply Quote 0
                    • B
                      Benoît Zuber @Ed Bartosh last edited by

                      @ed-bartosh UTC i guess. Well on some of the exchanges that I use they close at that midnight but I haven't thought that it might not be the case with all exchanges...

                      E 1 Reply Last reply Reply Quote 0
                      • E
                        Ed Bartosh @Benoît Zuber last edited by

                        @benoît-zuber

                        Wouldn't it make sense that the program only contacts the exchanges at midnight?

                        Can you elaborate on this a bit? How to reproduce this?

                        If ccxt plugin does this I can try to fix it. If it's backtrader, I'm not sure it's a good idea to implement this as it should work for all markets. Some of them are open at UTC midnight.

                        1 Reply Last reply Reply Quote 0
                        • B
                          Benoît Zuber last edited by

                          @Ed-Bartosh Yes you're right, I don't think ccxt knows when the transition between daily candles occurs for each exchange. The conservative approach of regularly fetching ohlcv without trying to guess when a new ohlcv occurs is safer.

                          1 Reply Last reply Reply Quote 0
                          • 1 / 1
                          • First post
                            Last post
                          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors