Anyone use backtrader to do live trading on Bitcoin exchange?
-
@sfkiwi yes, that issue was reported some time ago and there are 2 PRs about it: https://github.com/bartosh/backtrader/pull/13 and https://github.com/bartosh/backtrader/pull/6
Unfortunately they're both outdated and require rebase. Can you check if any of them fixes the issue for you?
-
Currently backfill_from is not supported. I'll be happy to see this implemented. I hope it's not that hard to implement considering working example from ibdata.py.
will add it to my backlog developments :)
can you maybe open the option for 'issues' in your repo. I have a couple more ideas for developments and maybe other people also would like to participate. ccxt topic is very active and using this one super long thread is messy
-
@jacob > can you maybe open the option for 'issues' in your repo.
done
-
Has anyone managed to fix sync/ gathering partial candle data issue mentioned above?
If the solution is here https://github.com/bartosh/backtrader/pull/6, why cannot we merge ? (github noob here, sorry :p). Btw, thank you @Ed-Bartosh for this fabulous work :) -
Hi @ed-bartosh!
I tried to run on bitmex exchange live:
data = bt.feeds.CCXT(exchange='bitmex', symbol='BTC/USD', timeframe=bt.TimeFrame.Ticks, compression=1)
When I tried to run live using Bitmex exchange I got some errors:
I open a issue (#20) and fixed on my repo fork.
Cloud you accept a pull request? My user is @dedeco on github. If not, the fix is on issue description.
Thanks!!
-
@dedeco > Cloud you accept a pull request?
Absolutely.
BTW, You don't need to ask permissions for this, just submit PR. In worse case scenario it'll be rejected :)
-
@totoro said in Anyone use backtrader to do live trading on Bitcoin exchange?:
if the solution is here https://github.com/bartosh/backtrader/pull/6, why cannot we merge ?
I believe comments in that PR explain why it's not merged.
Btw, thank you @Ed-Bartosh for this fabulous work :)
You're welcome. It's actually a community work, not just mine. Many people contributed to it.
BTW, you have chance to do it too :) -
Hello all, and thanks @Ed-Bartosh and others for your phenomenal work.
I am having a problem trading using backtrader on the Bittrex exchange (I have not tried other exchanges yet). The code below, slightly modified from Soren Pallesen's example 01-09-2018, places a limit order far from the current price to simply check the trading functionality:
from __future__ import (absolute_import, division, print_function, unicode_literals) import sys import time from datetime import datetime, timedelta import backtrader as bt import ccxt class TestStrategy(bt.Strategy): def start(self): self.counter = 0 print('START') def prenext(self): self.counter += 1 print('prenext len %d - counter %d' % (len(self), self.counter)) def __init__(self): self.dmi = bt.indicators.DMI(self.data, period=3) self.order = None def next(self): print('------ next len %d - counter %d' % (len(self), self.counter)) print(self.dmi.plusDI[0], self.dmi.minusDI[0], self.dmi.adx[0]) self.counter += 1 print('*' * 5, 'NEXT:', bt.num2date(self.data0.datetime[0]), self.data0._name, self.data0.open[0], self.data0.high[0], self.data0.low[0], self.data0.close[0], self.data0.volume[0], bt.TimeFrame.getname(self.data0._timeframe), len(self.data0)) if not self.getposition(data) and self.order==None: print('\nBUYING') self.order = self.buy(data, exectype=bt.Order.Limit, size=0.5, price=data.close[0]*0.5) print(self.order) else: print('\nSELLING') self.order = self.sell(data, exectype=bt.Order.Limit, size=0.5, price=data.close[0]*1.5) print(self.order) def notify_order(self, order): print('*' * 5, "NOTIFY ORDER", order) if __name__ == '__main__': cerebro = bt.Cerebro() exchange = 'bittrex' symbol = 'NEO/BTC' # Create broker broker_config = {'apiKey': 'xxxxxxxxxx', 'secret': 'xxxxxxxxxx', 'nonce': lambda: str(int(time.time() * 1000)) } broker = bt.brokers.CCXTBroker(exchange='bittrex', currency='BTC', config=broker_config) cerebro.setbroker(broker) # Create data feed hist_start_date = datetime.utcnow() - timedelta(minutes=30) data = bt.feeds.CCXT(exchange=exchange, symbol=symbol, timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date, compression=1) cerebro.adddata(data) cerebro.addstrategy(TestStrategy) cerebro.run()This successfully places an order on the exchange. However, the script crashes immediately afterward producing the following output:
START prenext len 1 - counter 1 prenext len 2 - counter 2 prenext len 3 - counter 3 prenext len 4 - counter 4 prenext len 5 - counter 5 ------ next len 6 - counter 5 45.64666519054272 24.016532585430657 32.95714573240973 ***** NEXT: 2018-09-03 15:40:00 0.00310706 0.00310706 0.00310706 0.00310706 0.5496506 Minute 6 BUYING Traceback (most recent call last): File "C:/Users/pns21/Documents/Python Files/Beginner and Tutuorial files/Backtrader - Bartosh/Peter Test 2018.09.03.py", line 83, in <module> cerebro.run() File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext strat._next() File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\strategy.py", line 325, in _next super(Strategy, self)._next() File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 268, in _next self.nextstart() # only called for the 1st value File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\lineiterator.py", line 342, in nextstart self.next() File "C:/Users/pns21/Documents/Python Files/Beginner and Tutuorial files/Backtrader - Bartosh/Peter Test 2018.09.03.py", line 45, in next price=data.close[0]*0.5) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\strategy.py", line 917, in buy **kwargs) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\brokers\ccxtbroker.py", line 86, in buy return self._submit(owner, data, exectype, 'buy', size, price, kwargs) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\brokers\ccxtbroker.py", line 78, in _submit order = CCXTOrder(owner, data, _order) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\metabase.py", line 88, in __call__ _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\metabase.py", line 78, in doinit _obj.__init__(*args, **kwargs) File "C:\Users\pns21\AppData\Local\Programs\Python\Python36\lib\site-packages\backtrader\stores\ccxtstore.py", line 39, in __init__ self.size = float(ccxt_order['amount']) TypeError: float() argument must be a string or a number, not 'NoneType'Thanks in advance for any help solving this issue!!
-
@petens said in Anyone use backtrader to do live trading on Bitcoin exchange?:
size=0.5
Does it work with size=1 ?
-
@ed-bartosh sadly it has the same result. The minimum order size on the bittrex exchange is 50,000 Satoshi or 0.0005 BTC, and either amount is over that for this symbol. Regardless the limit order is being successfully placed on the exchange. My inkling is that processing of the return values is causing this error, but i'm not experienced enough of a programmer to tell.
-
@Ed-Bartosh and others, I did some digging and I think I found the cause: the Bittrex API doesn't return any amount info.
In the process of making an order ccxtstore.py calls the ccxt method "create_order" (at line 119 in the current version) :order = self.exchange.create_order(symbol=symbol, type=order_type, side=side, amount=amount, price=price, params=params)This works and places an order on the exchange, and returns a dictionary to "order" like this:
print('order:', order) order: {'info': {'success': True, 'message': '', 'result': {'uuid': 'd9765e24-5d48-47b2-a343-4d487e679040'}}, 'id': 'd9765e24-5d48-47b2-a343-4d487e679040', 'symbol': 'NEO/BTC', 'type': 'limit', 'side': 'buy', 'status': 'open'}This is parsed with the ccxt method "parse_order" in the next command and returns a dict:
parsed_order = self.exchange.parse_order(order['info']) print('parsed_order:', parsed_order) parsed_order: {'info': {'success': True, 'message': '', 'result': {'uuid': 'd9765e24-5d48-47b2-a343-4d487e679040'}}, 'id': None, 'timestamp': None, 'datetime': None, 'lastTradeTimestamp': None, 'symbol': None, 'type': 'limit', 'side': None, 'price': None, 'cost': None, 'average': None, 'amount': None, 'filled': None, 'remaining': None, 'status': None, 'fee': None}It's obvious bittrex doesn't return much info about the order. This dict is passed to "ccxt_order" on the call of the CCXTOrder class, and the script crashes while running:
self.size = float(ccxt_order['amount'])from ccxtstore.py, because of the None value for the 'amount' key.
Bittrex doesn't give the necessary info but there's other ways to get it with their API. Any ideas on how to fix this? Should this be handled in the ccxt method "parse_order"?On another note, what other crypto exchanges work well for people using this fork of backtrader?
-
Regarding the partial candle data, even though the PR wasn't merged, you can pick up fork of the Ed's and the others awesome work here:
https://github.com/Dave-Vallance/bt-ccxt-store
The repo contains made a few changes I made including an attempt to address the partial candle issue (although the solution is different to the PR). These changes work for me on the few exchanges I am looking at but your mileage may vary and I don't really plan on supporting it :)
There are some other changes such as speeding up backfilling and allowing you to map some broker keys and strings if the defaults don't work amongst other things. (for example, since CCXT does not have a unified stop/limit order implementation, Kraken expects you to create a stoploss with 'stop-loss', whereas bitmex expects just 'stop'.
For a full list of changes you can checkout the readme.
-
@petens Can you check if this commit fixes the issue for you?
-
Hi,
I am trying to understand backtrader/ccxt/bitmex live behaviours.
I am running a test script : simple strat with bitmex data feed in 1 min bar and bitmex broker.
It computes a sma(period=2), waiting for this sma to be calculated and then send one single limit order.There is a delay I don't undertand:
If I load 4 bars from bitmex, without setting a broker, historical data are loading quickly and strategy run quickly.
With a broker it takes 1 min to run the strategy on 4 bars.
If I need few days of backfill, it will take forever. Is it normal ?@ThatBlokeDave : Is that the kind of problem you add and corrected when you say " speeding up backfilling" ?
Ouput without broker : = 5 sec to init
time is 2018-09-10 10:54:33.984922 ## start init pausetrade = True ***** DATA NOTIF: DELAYED time is 2018-09-10 10:54:38.298118 ## next ***** NEXT: 2018-09-10 08:51:00 6295.0 6295.0 6292.0 6292.5 365509.0 Minute 1 time is 2018-09-10 10:54:38.299116 ## next ***** NEXT: 2018-09-10 08:52:00 6292.5 6292.5 6290.5 6290.5 496485.0 Minute 2 time is 2018-09-10 10:54:38.300116 ## next ***** NEXT: 2018-09-10 08:53:00 6290.5 6290.5 6282.0 6282.0 2219366.0 Minute 3 time is 2018-09-10 10:54:38.300116 ## next ***** NEXT: 2018-09-10 08:54:00 6282.0 6284.0 6282.0 6284.0 2435976.0 Minute 4 ***** DATA NOTIF: LIVE time is 2018-09-10 10:55:03.080312 ## next ***** NEXT: 2018-09-10 08:55:00 6282.0 6282.0 6282.0 6282.0 0.0 Minute 5 time is 2018-09-10 10:55:03.080312 ## SEND BUY LIMIT: 1 @ 5283.00 Shutdown requested...exitingWith a broker : = 44 sec to init !!
time is 2018-09-10 12:13:18.374900 ## start init pausetrade = True ***** DATA NOTIF: DELAYED time is 2018-09-10 12:13:31.147005 ## next ***** NEXT: 2018-09-10 10:10:00 6279.0 6279.0 6278.5 6279.0 546301.0 Minute 1 time is 2018-09-10 12:13:41.502875 ## next ***** NEXT: 2018-09-10 10:11:00 6279.0 6279.5 6278.5 6279.5 335856.0 Minute 2 time is 2018-09-10 12:13:51.868567 ## next ***** NEXT: 2018-09-10 10:12:00 6279.5 6279.5 6279.0 6279.0 141525.0 Minute 3 time is 2018-09-10 12:14:02.329758 ## next ***** NEXT: 2018-09-10 10:13:00 6279.0 6279.5 6279.0 6279.0 21171.0 Minute 4 ***** DATA NOTIF: LIVE time is 2018-09-10 12:14:16.795837 ## next ***** NEXT: 2018-09-10 10:14:00 6279.5 6279.5 6279.0 6279.0 20132.0 Minute 5 time is 2018-09-10 12:14:20.980734 ## SEND BUY LIMIT: 1.0 @ 5279.00 ***** NOTIFY ORDER 0 time is 2018-09-10 12:15:08.587889 ## next ***** NEXT: 2018-09-10 10:15:00 6279.0 6279.0 6279.0 6279.0 0.0 Minute 6 Shutdown requested...exitingHere is my code for this test :
from datetime import datetime, timedelta import ccxt import backtrader as bt import os from pandas import bdate_range #get CCXT LOGS #import logging #logging.basicConfig(level=logging.DEBUG) class mmTest(bt.Strategy): params = (('pfast', 5), ('pslow', 30),) order = None pausetrade = True def log(self, txt, dt=None): ''' Logging function for this strategy''' dt = dt or self.datas[0].datetime.datetime(0) print('%s, %s' % (str(dt), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.pausetrade = not self.data.haslivedata() print( "init pausetrade = " , self.pausetrade) self.dataclose = self.datas[0].close self.mm = bt.ind.MovingAverageSimple(self.data.close,period=2) self.order = None def notify_order(self, order): print('*' * 5, "NOTIFY ORDER", order.status ) def notify_data(self, data, status, *args, **kwargs): self.pausetrade = (status != data.LIVE) print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args ) def sendBuyOrder(self): if self.pausetrade: return if self.mm[0] >0: limit=self.mm[0] - 1000 ticksize = 0.5 limit=round(limit/ticksize)*ticksize self.order =self.buy(price= limit, exectype=bt.Order.Limit,transmit=True) print('time is ',datetime.now(),' ## SEND BUY LIMIT: {:s} @ {:.2f}'.format(str(self.order.size),limit )) else: print('MM = %f' % (self.mm[0])) def next(self): print('time is ',datetime.now(),' ## next') for data in self.datas: print('*' * 5, 'NEXT:', bt.num2date(data.datetime[0]), data._name, data.open[0], data.high[0],data.low[0], data.close[0], data.volume[0], bt.TimeFrame.getname(data._timeframe), len(data)) if not self.order : self.sendBuyOrder() try: print('time is ',datetime.now(),' ## start') cerebro = bt.Cerebro() feedconf = {'apiKey': '8KoTAFQZ3G1zkgRPSjj-LZi3', 'secret': ''} data = bt.feeds.CCXT(exchange="bitmex", symbol="BTC/USD",fromdate=datetime.now()+timedelta(hours = -2,minutes=-4), timeframe=bt.TimeFrame.Minutes,compression=1,config=feedconf,backfill_start=True) broker_config = {'apiKey': '8KoTAFQZ3G1zkgRPSjj-LZi3', 'secret': '****',} broker = bt.brokers.CCXTBroker(exchange='bitmex', currency='XBt', config=broker_config) cerebro.setbroker(broker) cerebro.adddata(data) cerebro.addstrategy(mmTest) cerebro.run() #cerebro.plot() except KeyboardInterrupt: print("Shutdown requested...exiting") #except Exception: # traceback.print_exc(file=sys.stdout) quit -
Hey @ed-bartosh, that does seem to do the trick, a million thanks! I'll keep looking for issues and fixes to contribute as I go along using this.
-
@søren-pallesen
Hi,
Did you get correct speed for backfilling few minutes bars when you use Bitmex feed AND Bitmex broker?
It is seems that 1-minute bars are received in one block but a simple strategy with 1 indicator needs at least 10sec per bar to run/compute the "next" function.May be, it is the normal behaviour of Backtrader running live?
-
Hi, not sure if this is the right place to ask but how do you input backfeed data from ccxt into backtrader? Do I have to convert the ccxt output to CSV and proceed from there?
import ccxt bitmex = ccxt.bitmex() # params: symbol = 'BTC/USD' timeframe = '1d' limit = 750 params = {'partial': False} # ←-------- no reversal since = None candles = bitmex.fetch_ohlcv(symbol, timeframe, since, limit, params)Here I want to use the data in candles as feed for backtesting (not live trading). Thank you!
-
@thomaslee
You can probably convert candles into a Pandas dataframe with datetime, open, high, low , close, volume.
And then use this feed :data0=bt.feeds.PandasDirectData(dataname=dataframe,**kwargs,datetime=0,time=1,open=2,high=3,low=4,close=5,volume=6,openinterest=-1,timeframe=bt.TimeFrame.Days,compression=1) -
Is that the kind of problem you add and corrected when you say " speeding up backfilling" ?
Yes - That is correct. The reason it is slow is because the broker will try to check your cash/balance on every call of
next(). (If I remember correctly it makes 2 or 3 checks in total between each call of next). Since, the store will make use rest API calls to do these checks, this can add quite a few seconds delay between each call ofnext(). (Because of the rest API rate limit).So in my fork, I cheat a bit. The cash and balance checks return the last values received from the last rest call. They do not make another call. Instead, the user must manually check the balance if they need up to date balance information before entering a position. (at which point the latest values will be saved).
Not an ideal solution but it works for me.
-
@ed-bartosh said in Anyone use backtrader to do live trading on Bitcoin exchange?:
guide
Do you ever experience the problem of self.order = self.buy(size=1.5) or in the buy method ,the param size can not set to a float value ?
