For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
dealing with ExchangeError in backtrader ccxt branch
-
@Ed-Bartosh Hi, congrats for this branch, it is fantastic! I was just just wondering how you deal with ExchangeError that occur randomly from time to time. Namely bittrex api sometimes returns an empty reply and cerebro.run() crashes. Is there a trick to catch the exception and ignore it without touching the source code of backtrader or ccxt?
Here is my dummy example:class TestStrategy(bt.Strategy): def next(self): print("time now: %s." %time.strftime("%H:%M:%S", time.gmtime())) for data in self.datas: print(bt.num2date(data.datetime[0]), data._name, data.open[0], data.high[0], data.low[0], data.close[0], data.volume[0], "day", len(data)) def runstrategy(): # Create a cerebro cerebro = bt.Cerebro() # Create a broker hist_start_date = datetime.utcnow() - timedelta(days=10) #broker = bt.brokers.CCXTBroker(exchange='bittrex') data_min1 = bt.feeds.CCXT(exchange = 'bittrex', symbol='BTC/USDT', name="btc_usdt_bittrex", timeframe=bt.TimeFrame.Days, compression = 1, fromdate = hist_start_date) data_min2 = bt.feeds.CCXT(exchange = 'gdax', symbol='BTC/USD', name="btc_usdt_gdax", timeframe=bt.TimeFrame.Days, compression = 1, fromdate = hist_start_date) cerebro.adddata(data_min1) cerebro.adddata(data_min2) cerebro.addstrategy(TestStrategy) cerebro.run()When I run strategy() I am getting this output:
time now: 21:33:20. 2018-02-10 00:00:00 btc_usdt_bittrex 8689.0 9041.40405 8140.0 8540.0 6842.4350137 day 1 2018-02-10 00:00:00 btc_usdt_gdax 8671.0 9090.0 8155.0 8547.49 6763.749508230131 day 1 time now: 21:33:20. 2018-02-11 00:00:00 btc_usdt_bittrex 8540.00000001 8551.0 7815.73000001 8052.5 4819.85510126 day 2 2018-02-11 00:00:00 btc_usdt_gdax 8547.48 8547.49 7851.0 8072.99 5030.079040800108 day 2 time now: 21:33:20. 2018-02-12 00:00:00 btc_usdt_bittrex 8052.5 8980.0 8052.0 8913.48826283 4337.03012567 day 3 2018-02-12 00:00:00 btc_usdt_gdax 8073.0 8950.0 8072.99 8872.28 5192.890664420066 day 3 time now: 21:33:20. 2018-02-13 00:00:00 btc_usdt_bittrex 8913.48826283 8945.0 8359.0 8512.00000005 3382.11671751 day 4 2018-02-13 00:00:00 btc_usdt_gdax 8872.27 8925.0 8393.1 8520.01 4254.992621180045 day 4 time now: 21:33:20. 2018-02-14 00:00:00 btc_usdt_bittrex 8512.00000005 9487.6 8512.00000004 9450.0 4072.19201411 day 5 2018-02-14 00:00:00 btc_usdt_gdax 8520.01 9482.5 8520.0 9472.98 6591.748869050115 day 5 time now: 21:33:20. 2018-02-15 00:00:00 btc_usdt_bittrex 9450.0 10190.00000002 9330.0 10035.00000004 5343.13567805 day 6 2018-02-15 00:00:00 btc_usdt_gdax 9472.98 10249.81 9342.46 10031.23 7688.429391570176 day 6 time now: 21:33:20. 2018-02-16 00:00:00 btc_usdt_bittrex 10035.00000004 10287.0 9660.0 10152.0 3018.51905787 day 7 2018-02-16 00:00:00 btc_usdt_gdax 10027.8 10307.68 9730.01 10167.49 5089.095234060047 day 7 time now: 21:33:20. 2018-02-17 00:00:00 btc_usdt_bittrex 10150.0 11098.0 10049.00000201 11048.4 4079.41257877 day 8 2018-02-17 00:00:00 btc_usdt_gdax 10167.49 11149.0 10057.0 11121.5 5717.07199700005 day 8 time now: 21:33:20. 2018-02-18 00:00:00 btc_usdt_bittrex 11048.99999998 11245.0 10080.99999998 10380.00000009 5746.00081324 day 9 2018-02-18 00:00:00 btc_usdt_gdax 11121.5 11299.1 10159.0 10380.04 6455.1921786601015 day 9 time now: 21:33:20. 2018-02-19 00:00:00 btc_usdt_bittrex 10380.00000009 11238.23760818 10303.45443897 11130.0 3365.1113783 day 10 2018-02-19 00:00:00 btc_usdt_gdax 10380.04 11274.11 10297.39 11102.5 4042.6038281900164 day 10 time now: 00:01:02. 2018-02-19 00:00:00 btc_usdt_bittrex 10380.00000009 11238.23760818 10303.45443897 11130.0 3365.1113783 day 10 2018-02-20 00:00:00 btc_usdt_gdax 11140.0 11140.0 11139.99 11140.0 1.67212084 day 11 time now: 00:02:40. 2018-02-20 00:00:00 btc_usdt_bittrex 11159.999 11160.0 11159.999 11159.999 0.6325051 day 11 2018-02-20 00:00:00 btc_usdt_gdax 11140.0 11140.0 11139.99 11140.0 1.67212084 day 11 --------------------------------------------------------------------------- ExchangeError Traceback (most recent call last) <ipython-input-5-8a8200ffe2f4> in <module>() 1 #th = Thread(target = runstrategy) 2 #th.start() ----> 3 runstrategy() <ipython-input-4-45bb8d84d254> in runstrategy() 17 cerebro.adddata(data_min2) 18 cerebro.addstrategy(TestStrategy) ---> 19 cerebro.run() ~/anaconda3/lib/python3.6/site-packages/backtrader/cerebro.py in run(self, **kwargs) 1125 # let's skip process "spawning" 1126 for iterstrat in iterstrats: -> 1127 runstrat = self.runstrategies(iterstrat) 1128 self.runstrats.append(runstrat) 1129 else: ~/anaconda3/lib/python3.6/site-packages/backtrader/cerebro.py in runstrategies(self, iterstrat, predata) 1293 self._runnext_old(runstrats) 1294 else: -> 1295 self._runnext(runstrats) 1296 1297 for strat in runstrats: ~/anaconda3/lib/python3.6/site-packages/backtrader/cerebro.py in _runnext(self, runstrats) 1536 qlapse = datetime.datetime.utcnow() - qstart 1537 d.do_qcheck(newqcheck, qlapse.total_seconds()) -> 1538 drets.append(d.next(ticks=False)) 1539 1540 d0ret = any((dret for dret in drets)) ~/anaconda3/lib/python3.6/site-packages/backtrader/feed.py in next(self, datamaster, ticks) 402 403 # not preloaded - request next bar --> 404 ret = self.load() 405 if not ret: 406 # if load cannot produce bars - forward the result ~/anaconda3/lib/python3.6/site-packages/backtrader/feed.py in load(self) 474 475 if not self._fromstack(stash=True): --> 476 _loadret = self._load() 477 if not _loadret: # no bar use force to make sure in exactbars 478 # the pointer is undone this covers especially (but not ~/anaconda3/lib/python3.6/site-packages/backtrader/feeds/ccxt.py in _load(self) 89 return self._load_ticks() 90 else: ---> 91 self._fetch_ohlcv() 92 return self._load_ohlcv() 93 elif self._state == self._ST_HISTORBACK: ~/anaconda3/lib/python3.6/site-packages/backtrader/feeds/ccxt.py in _fetch_ohlcv(self, fromdate) 123 dlen = len(self._data) 124 for ohlcv in sorted(self.store.fetch_ohlcv(self.symbol, timeframe=granularity, --> 125 since=since, limit=limit)): 126 if None in ohlcv: 127 continue ~/anaconda3/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py in retry_method(self, *args, **kwargs) 85 time.sleep(self.exchange.rateLimit / 1000) 86 try: ---> 87 return method(self, *args, **kwargs) 88 except NetworkError: 89 if i == self.retries - 1: ~/anaconda3/lib/python3.6/site-packages/backtrader/stores/ccxtstore.py in fetch_ohlcv(self, symbol, timeframe, since, limit) 119 @retry 120 def fetch_ohlcv(self, symbol, timeframe, since, limit): --> 121 return self.exchange.fetch_ohlcv(symbol, timeframe=timeframe, since=since, limit=limit) 122 123 @retry ~/anaconda3/lib/python3.6/site-packages/ccxt/bittrex.py in fetch_ohlcv(self, symbol, timeframe, since, limit, params) 402 if response['result']: 403 return self.parse_ohlcvs(response['result'], market, timeframe, since, limit) --> 404 raise ExchangeError(self.id + ' returned an empty or unrecognized response: ' + self.json(response)) 405 406 def fetch_open_orders(self, symbol=None, since=None, limit=None, params={}): ExchangeError: bittrex returned an empty or unrecognized response: {"success":true,"message":"","result":null} -
@benoît-zuber I'm afraid ExchangeError can't be avoided in the strategy code. However, it might make sense to do in ccxt store. Can you try if this patch fixes the issue for you?
diff --git a/backtrader/stores/ccxtstore.py b/backtrader/stores/ccxtstore.py index 16f6406..b58e0a0 100644 --- a/backtrader/stores/ccxtstore.py +++ b/backtrader/stores/ccxtstore.py @@ -25,7 +25,7 @@ import time from functools import wraps import ccxt -from ccxt.base.errors import NetworkError +from ccxt.base.errors import NetworkError, ExchangeError import backtrader as bt @@ -85,7 +85,7 @@ class CCXTStore(object): time.sleep(self.exchange.rateLimit / 1000) try: return method(self, *args, **kwargs) - except NetworkError: + except NetworkError, ExchangeError: if i == self.retries - 1: raise -
@ed-bartosh Sure I am going to try. How can you specify the maximum number of retries? From my previous experience it needs to retry for about one minute in some cases.
-
@benoît-zuber to specify number of retries you can use 'retries' feed parameter.
-
@Ed-Bartosh will this fix get merged into your main CCXT branch?
-
@søren-pallesen Yes, I'm going to merge it after @Benoît-Zuber confirms that it works.
-
@Ed-Bartosh I have tested your patch and it works perfectly. I added a print statement at in the except block. I paste below a part of the output. Note that I set retries to 100 in the feeds.
2018-02-27, BUY PURA/BTC: size is 1852.797522, value is 0.148205. 2018-02-27, o:7.999e-05, h:8e-05, l:6.8e-05, c:6.82e-05 2018-02-27, previous highest close : 7.999e-05, atr7.407824056208675e-06 2018-02-27, OPERATION PROFIT for LTC/BTC, GROSS -0.03, NET -0.03, ACCOUNT VALUE 0.92622656 2018-02-27, prenext 2018-02-27, 1 open trade(s): PURA/BTC. retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 6; maximum retries: 100 retrying, i = 7; maximum retries: 100 retrying, i = 8; maximum retries: 100 retrying, i = 9; maximum retries: 100 retrying, i = 10; maximum retries: 100 retrying, i = 11; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 6; maximum retries: 100 retrying, i = 7; maximum retries: 100 retrying, i = 8; maximum retries: 100 retrying, i = 9; maximum retries: 100 retrying, i = 10; maximum retries: 100 retrying, i = 11; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 6; maximum retries: 100 retrying, i = 7; maximum retries: 100 retrying, i = 8; maximum retries: 100 retrying, i = 9; maximum retries: 100 retrying, i = 10; maximum retries: 100 retrying, i = 11; maximum retries: 100 retrying, i = 12; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 6; maximum retries: 100 retrying, i = 7; maximum retries: 100 retrying, i = 8; maximum retries: 100 retrying, i = 9; maximum retries: 100 retrying, i = 10; maximum retries: 100 retrying, i = 11; maximum retries: 100 retrying, i = 12; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 6; maximum retries: 100 retrying, i = 7; maximum retries: 100 retrying, i = 8; maximum retries: 100 retrying, i = 9; maximum retries: 100 retrying, i = 10; maximum retries: 100 retrying, i = 11; maximum retries: 100 retrying, i = 12; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 6; maximum retries: 100 retrying, i = 7; maximum retries: 100 retrying, i = 8; maximum retries: 100 retrying, i = 9; maximum retries: 100 retrying, i = 10; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 1; maximum retries: 100 retrying, i = 2; maximum retries: 100 retrying, i = 3; maximum retries: 100 retrying, i = 4; maximum retries: 100 retrying, i = 5; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 retrying, i = 0; maximum retries: 100 2018-02-27, prenext 2018-02-27, 1 open trade(s): PURA/BTC.I just had to put NetworkError and ExchangeError in a tuple at the except line. I am using python 3.6.
BTW It seems that backtrader keeps asking the exchanges for the latest daily candle. This seems like a waste of resources. Wouldn't it make sense that the program only contacts the exchanges at midnight?
-
@benoît-zuber said in dealing with ExchangeError in backtrader ccxt branch:
Wouldn't it make sense that the program only contacts the exchanges at midnight?
Midnight in which time zone?
-
@ed-bartosh UTC i guess. Well on some of the exchanges that I use they close at that midnight but I haven't thought that it might not be the case with all exchanges...
-
Wouldn't it make sense that the program only contacts the exchanges at midnight?
Can you elaborate on this a bit? How to reproduce this?
If ccxt plugin does this I can try to fix it. If it's backtrader, I'm not sure it's a good idea to implement this as it should work for all markets. Some of them are open at UTC midnight.
-
@Ed-Bartosh Yes you're right, I don't think ccxt knows when the transition between daily candles occurs for each exchange. The conservative approach of regularly fetching ohlcv without trying to guess when a new ohlcv occurs is safer.