Anyone use backtrader to do live trading on Bitcoin exchange?
-
@mr-m0nst3r said in Anyone use backtrader to do live trading on Bitcoin exchange?:
@ed-bartosh
Hi Bartosh,Thank you for the implementation of ccxt in backtrader.
I'm trying to add 1 minute data to feed, and then resample it to 2 minutes data, so I can get MACD of 1 minute and MACD of 2 minutes in the Strategy.
But I'm keeping getting the error of:
ValueError: min() arg is an empty sequence
Here's the code:
# !/usr/bin/env python # -*- coding: utf-8; py-indent-offset:4 -*- from __future__ import (absolute_import, division, print_function, unicode_literals) import sys import time from datetime import datetime, timedelta from datetime import datetime, timedelta import backtrader as bt import ccxt class TestStrategy(bt.Strategy): params = ( ('printlog', True), ) def log(self, txt, dt=None, doprint=False): ''' Logging function fot this strategy''' if self.params.printlog or doprint: dt = dt or bt.num2date(self.data.datetime[0]) print('%s, %s' % (dt, txt)) def start(self): self.counter = 0 print('START') def prenext(self): self.counter += 1 print('prenext len %d - counter %d' % (len(self), self.counter)) def __init__(self): self.macd = bt.indicators.MACDHisto(self.datas[0]) self.macd2 = bt.indicators.MACDHisto(self.datas[1]) def next(self): self.counter += 1 price_txt = "Counter: " + str(self.counter) + " Open/Close/High/Low/Volume: " + str(self.data0.open[0]) + " - "+ str(self.data0.close[0]) + " - " + str(self.data0.high[0]) + " - " + str(self.data0.low[0]) + " - " + str(self.data0.volume[0]) + " Len: "+ str(len(self.data0))# + " Time Frame:" + bt.TimeFrame.getname(self.data0._timeframe) + " Len: "+ str(len(self.data0)) self.log(price_txt) macd_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd.macd[0],self.macd.histo[0]) self.log("MACD#1: " + macd_txt) macd2_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd2.macd[0],self.macd2.histo[0]) self.log("MACD#2: " + macd2_txt) if __name__ == '__main__': cerebro = bt.Cerebro() #exchange = sys.argv[1] if len(sys.argv) > 1 else 'gdax' exchange = sys.argv[1] if len(sys.argv) > 1 else 'okex' symbol = sys.argv[2] if len(sys.argv) > 2 else 'ETH/USDT' hist_start_date = datetime.utcnow() - timedelta(minutes=10) print('UTC NOW: ', datetime.utcnow()) print('hist_start_data: ', hist_start_date) print('Using symbol: ', symbol) # Create data feeds data_1m = bt.feeds.CCXT(exchange=exchange, symbol=symbol, name="1m", timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date,compression=1) cerebro.adddata(data_1m) cerebro.resampledata(data_1m, timeframe=bt.TimeFrame.Minutes, compression=2) cerebro.addstrategy(TestStrategy) cerebro.run()
The output is:
python stage1.py UTC NOW: 2019-06-21 05:41:48.784742 hist_start_data: 2019-06-21 05:31:48.784520 Using symbol: ETH/USDT START prenext len 1 - counter 1 prenext len 2 - counter 2 prenext len 3 - counter 3 prenext len 4 - counter 4 prenext len 5 - counter 5 prenext len 6 - counter 6 prenext len 7 - counter 7 prenext len 8 - counter 8 prenext len 9 - counter 9 prenext len 10 - counter 10 prenext len 11 - counter 11 prenext len 12 - counter 12 Traceback (most recent call last): File "stage1.py", line 66, in <module> cerebro.run() File "/Users/michael/.venv/backtrader-ccxt/lib/python3.6/site-packages/backtrader/cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "/Users/michael/.venv/backtrader-ccxt/lib/python3.6/site-packages/backtrader/cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "/Users/michael/.venv/backtrader-ccxt/lib/python3.6/site-packages/backtrader/cerebro.py", line 1557, in _runnext dt0 = min((d for i, d in enumerate(dts) ValueError: min() arg is an empty sequence
What am I doing wrong here?
Thank you.
I changed
backtrader.cerebro.py
to print thedts
and changed strategy to print theself.datas[i].datetime[0]
, and here's the output:python stage1.py UTC NOW: 2019-06-21 08:33:35.360573 hist_start_data: 2019-06-21 08:23:35.360331 Using symbol: ETH/USDT START dts[]: [737231.35, 737231.35] prenext len 1 - counter 1 2019-06-21 08:24:00, data#0: 737231.35 2019-06-21 08:24:00, data#1: 737231.35 dts[]: [737231.3506944445, None] prenext len 2 - counter 2 2019-06-21 08:25:00, data#0: 737231.3506944445 2019-06-21 08:25:00, data#1: 737231.35 dts[]: [737231.3513888889, 737231.3513888889] prenext len 3 - counter 3 2019-06-21 08:26:00, data#0: 737231.3513888889 2019-06-21 08:26:00, data#1: 737231.3513888889 dts[]: [737231.3520833333, None] prenext len 4 - counter 4 2019-06-21 08:27:00, data#0: 737231.3520833333 2019-06-21 08:27:00, data#1: 737231.3513888889 dts[]: [737231.3527777778, 737231.3527777778] prenext len 5 - counter 5 2019-06-21 08:28:00, data#0: 737231.3527777778 2019-06-21 08:28:00, data#1: 737231.3527777778 dts[]: [737231.3534722222, None] prenext len 6 - counter 6 2019-06-21 08:29:00, data#0: 737231.3534722222 2019-06-21 08:29:00, data#1: 737231.3527777778 dts[]: [737231.3541666666, 737231.3541666666] prenext len 7 - counter 7 2019-06-21 08:30:00, data#0: 737231.3541666666 2019-06-21 08:30:00, data#1: 737231.3541666666 dts[]: [737231.3548611111, None] prenext len 8 - counter 8 2019-06-21 08:31:00, data#0: 737231.3548611111 2019-06-21 08:31:00, data#1: 737231.3541666666 dts[]: [737231.3555555556, 737231.3555555556] prenext len 9 - counter 9 2019-06-21 08:32:00, data#0: 737231.3555555556 2019-06-21 08:32:00, data#1: 737231.3555555556 dts[]: [737231.35625, None] prenext len 10 - counter 10 2019-06-21 08:33:00, data#0: 737231.35625 2019-06-21 08:33:00, data#1: 737231.3555555556 dts[]: [None, 737231.3569444445] Traceback (most recent call last): File "stage1.py", line 69, in <module> cerebro.run() File "/Users/michael/.venv/backtrader-ccxt/lib/python3.6/site-packages/backtrader/cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "/Users/michael/.venv/backtrader-ccxt/lib/python3.6/site-packages/backtrader/cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "/Users/michael/.venv/backtrader-ccxt/lib/python3.6/site-packages/backtrader/cerebro.py", line 1558, in _runnext dt0 = min((d for i, d in enumerate(dts) ValueError: min() arg is an empty sequence
Hope it's helping to debug it.
@backtrader , is it the
cerebro.py
's problem?
the strategy is trying toresample
a live feed from ccxt.
Need help. -
Hi, I am trying to backtest the following basic strategy:
(It would buy when the RSI<30 and sell when the RSI>30)
future__ import(absolute_import, division, print_function, unicode_literals) from datetime import datetime, timedelta import backtrader as bt from backtrader import cerebro import time class firstStrategy(bt.Strategy): def __init__(self): self.rsi = bt.indicators.RSI_SMA(self.data.close, period=21) def next(self): if not self.position: if self.rsi < 30: self.buy(size=100) else: if self.rsi > 70: self.sell(size=100) #Variable for our starting cash startcash = 10000 if __name__ == '__main__': cerebro = bt.Cerebro() hist_start_date = datetime.utcnow() - timedelta(minutes=1000) data_min = bt.feeds.CCXT(exchange='binance', symbol="BTC/USDT", name="btc_usd_min", fromdate=hist_start_date, todate=datetime.utcnow(), timeframe=bt.TimeFrame.Minutes ) cerebro.adddata(data_min) cerebro.broker.setcash(startcash) cerebro.addstrategy(firstStrategy) cerebro.run() # Get final portfolio Value portvalue = cerebro.broker.getvalue() pnl = portvalue - startcash # Print out the final result print('Final Portfolio Value: ${}'.format(portvalue)) print('P/L: ${}'.format(pnl)) # Finally plot the end results cerebro.plot(style='candlestick')
The issue that I find is that it never buys or sell even when there are RSI values <30 and>70.
This is the result that I get:
Final Portfolio Value: $10000.0
P/L: $0.0And the chart is this (no buys or sells):
Please, help! what am I doing wrong?
Thanks!!
-
@mariano-bengeldorf Did you try using a smaller size. It seems you can not buy with size=100 at a price of about 1000$ per BTC?!
-
Hi @cptanpanic
I am live trading since a few weeks on Bitfinex. Seems to work now after a few hickups. I'm trading with a single data feed/currency on the daily base. Bot sending me emails and telegram messages on all decisions and events.
Used a boilerplate from Rodrigo?? not using the BUY/SELL sentinels. As well got trailing and normal stop loss orders on Bitfinex working canceling them before buying or selling. Want to make it nicer and more generic with arguments like strategy to use to be parsed etc.... and how to use a systemd service on Linux to run the strategy or have it run in a screen... and finally make it available here as a starting point/boilerplate to say thanks to all the superbe people in this forum having made this possible! -
@planet-winter Thank you so much!!!!
You are the best!!
One more question, the script never stops running, I have to hit ctrl+C to stop it. Do you know why that might be?Thanks
-
hi @mariano-bengeldorf. When plotting the open plot is a process stemming from your console script. Closing the plot will end the script as well. If this is not the problem or in general you can try backtesting with CSV files for example. They are certainly stopping your data feed and thus your script. Makes sense if you want to backtest over bigger time frames as well, having the data ready without loading data from the web API all the time. There are as well parameters to the ccxt feeds I don't remember as I don have my code ready. historical=? or so to ?
-
@planet-winter said in Anyone use backtrader to do live trading on Bitcoin exchange?:
hi @mariano-bengeldorf. When plotting the open plot is a process stemming from your console script. Closing the plot will end the script as well. If this is not the problem or in general you can try backtesting with CSV files for example. They are certainly stopping your data feed and thus your script. Makes sense if you want to backtest over bigger time frames as well, having the data ready without loading data from the web API all the time. There are as well parameters to the ccxt feeds I don't remember as I don have my code ready. historical=? or so to ?
Thanks @Planet-Winter , closing the plot doesn't stop the script. Will try the second alternative. Many thanks
-
Hi everybody!!
I am trying to backtest the following basic strategy:(It would buy when the RSI<30 and sell when the RSI>30)
I am having some issues with the analyzer
from __future__ import(absolute_import, division, print_function, unicode_literals) from datetime import datetime, timedelta import backtrader as bt from collections import OrderedDict from backtrader import cerebro import time class firstStrategy(bt.Strategy): def __init__(self): self.rsi = bt.indicators.RSI_SMA(self.data.close, period=21) def next(self): if not self.position: if self.rsi < 30: self.buy(size=0.1) else: if self.rsi > 70: self.sell(size=0.1) def printTradeAnalysis(analyzer): ''' Function to print the Technical Analysis results in a nice format. ''' #Get the results we are interested in total_open = analyzer.total.open total_closed = analyzer.total.closed total_won = analyzer.won.total total_lost = analyzer.lost.total win_streak = analyzer.streak.won.longest lose_streak = analyzer.streak.lost.longest pnl_net = round(analyzer.pnl.net.total,2) strike_rate = (total_won / total_closed) * 100 #Designate the rows h1 = ['Total Open', 'Total Closed', 'Total Won', 'Total Lost'] h2 = ['Strike Rate','Win Streak', 'Losing Streak', 'PnL Net'] r1 = [total_open, total_closed,total_won,total_lost] r2 = [strike_rate, win_streak, lose_streak, pnl_net] #Check which set of headers is the longest. if len(h1) > len(h2): header_length = len(h1) else: header_length = len(h2) #Print the rows print_list = [h1,r1,h2,r2] row_format ="{:<15}" * (header_length + 1) print("Trade Analysis Results:") for row in print_list: print(row_format.format('',*row)) def printSQN(analyzer): sqn = round(analyzer.sqn,2) print('SQN: {}'.format(sqn)) #Variable for our starting cash startcash = 100000 if __name__ == '__main__': cerebro = bt.Cerebro() hist_start_date = datetime.utcnow() - timedelta(minutes=1000) data_min = bt.feeds.CCXT(exchange='binance', symbol="BTC/USDT", dataname="btc_usd_min", fromdate=hist_start_date, todate=datetime.utcnow(), timeframe=bt.TimeFrame.Minutes ) cerebro.adddata(data_min) cerebro.broker.setcash(startcash) cerebro.addstrategy(firstStrategy) cerebro.run() # Add the analyzers we are interested in cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name="ta") cerebro.addanalyzer(bt.analyzers.SQN, _name="sqn") # Run over everything strategies = cerebro.run() firstStrat = strategies[0] # print the analyzers printTradeAnalysis(firstStrat.analyzers.ta.get_analysis()) printSQN(firstStrat.analyzers.sqn.get_analysis()) # Get final portfolio Value portvalue = cerebro.broker.getvalue() pnl = portvalue - startcash # Print out the final result print('Final Portfolio Value: ${}'.format(portvalue)) print('P/L: ${}'.format(pnl)) # Finally plot the end results cerebro.plot(style='candlestick')
I am getting the following error:
Traceback (most recent call last): File "rsi.py", line 114, in <module> printTradeAnalysis(firstStrat.analyzers.ta.get_analysis()) File "rsi.py", line 54, in printTradeAnalysis total_open = analyzer.total.open File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/utils/autodict.py", line 104, in __getattr__ return self[key] File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/utils/autodict.py", line 94, in __missing__ raise KeyError KeyError
Does anybody know what it means?
Many thanks in advance!! -
@mariano-bengeldorf: It means that there is not enough (buy/sell) data in your 'firstStrat.analyzers.ta.get_analysis()' to do the analysis/statstics on.
-
@itsmi said in Anyone use backtrader to do live trading on Bitcoin exchange?:
@mariano-bengeldorf: It means that there is not enough (buy/sell) data in your 'firstStrat.analyzers.ta.get_analysis()' to do the analysis/statstics on.
Thanks @itsmi . I have followed your suggestion and added 365224*60 (meaning two years in minutes) and now I get this other error message:
Traceback (most recent call last): File "rsi.py", line 103, in <module> cerebro.run() File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/cerebro.py", line 1630, in _runnext strat._next() File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/strategy.py", line 325, in _next super(Strategy, self)._next() File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/lineiterator.py", line 258, in _next indicator._next() File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/lineiterator.py", line 258, in _next indicator._next() File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/linebuffer.py", line 619, in _next self.next() File "/anaconda3/envs/ccxt/lib/python3.7/site-packages/backtrader/linebuffer.py", line 745, in next self[0] = self.operation(self.a[0], self.b[0]) ZeroDivisionError: float division by zero
Any advise?
Thanks again
-
Hi again...
Does anybody has a code to download minute data from BTC/USD since 2017/01/01 to a CVS. I am using the following, but it only downloads 14000 data points:import ccxt from datetime import datetime, timedelta, timezone import math import argparse import pandas as pd import ciso8601 import time def getUNIX(input_date): UNIX = int(time.mktime(ciso8601.parse_datetime(input_date).timetuple()) * 1000) return UNIX def parse_args(): parser = argparse.ArgumentParser(description='CCXT Market Data Downloader') parser.add_argument('-s','--symbol', type=str, required=True, help='The Symbol of the Instrument/Currency Pair To Download') parser.add_argument('-e','--exchange', type=str, required=True, help='The exchange to download from') parser.add_argument('-t','--timeframe', type=str, default='1d', choices=['1m', '5m','15m', '30m','1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1y'], help='The timeframe to download') parser.add_argument('--debug', action ='store_true', help=('Print Sizer Debugs')) return parser.parse_args() # Get our arguments args = parse_args() # Get our Exchange try: exchange = getattr (ccxt, args.exchange) () except AttributeError: print('-'*36,' ERROR ','-'*35) print('Exchange "{}" not found. Please check the exchange is supported.'.format(args.exchange)) print('-'*80) quit() # Check if fetching of OHLC Data is supported if exchange.has["fetchOHLCV"] != True: print('-'*36,' ERROR ','-'*35) print('{} does not support fetching OHLC data. Please use another exchange'.format(args.exchange)) print('-'*80) quit() # Check requested timeframe is available. If not return a helpful error. if (not hasattr(exchange, 'timeframes')) or (args.timeframe not in exchange.timeframes): print('-'*36,' ERROR ','-'*35) print('The requested timeframe ({}) is not available from {}\n'.format(args.timeframe,args.exchange)) print('Available timeframes are:') for key in exchange.timeframes.keys(): print(' - ' + key) print('-'*80) quit() # Check if the symbol is available on the Exchange exchange.load_markets() if args.symbol not in exchange.symbols: print('-'*36,' ERROR ','-'*35) print('The requested symbol ({}) is not available from {}\n'.format(args.symbol,args.exchange)) print('Available symbols are:') for key in exchange.symbols: print(' - ' + key) print('-'*80) quit() # Get data since = getUNIX('2017-01-01') data = exchange.fetch_ohlcv(args.symbol, args.timeframe, since) header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume'] df = pd.DataFrame(data, columns=header).set_index('Timestamp') # Save it symbol_out = args.symbol.replace("/","") filename = '{}-{}-{}.csv'.format(args.exchange, symbol_out,args.timeframe) df.to_csv(filename)
Thanks
-
@mariano-bengeldorf said in Anyone use backtrader to do live trading on Bitcoin exchange?:
Hi again...
Does anybody has a code to download minute data from BTC/USD since 2017/01/01 to a CVS. I am using the following, but it only downloads 14000 data points:import ccxt from datetime import datetime, timedelta, timezone import math import argparse import pandas as pd import ciso8601 import time def getUNIX(input_date): UNIX = int(time.mktime(ciso8601.parse_datetime(input_date).timetuple()) * 1000) return UNIX def parse_args(): parser = argparse.ArgumentParser(description='CCXT Market Data Downloader') parser.add_argument('-s','--symbol', type=str, required=True, help='The Symbol of the Instrument/Currency Pair To Download') parser.add_argument('-e','--exchange', type=str, required=True, help='The exchange to download from') parser.add_argument('-t','--timeframe', type=str, default='1d', choices=['1m', '5m','15m', '30m','1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1y'], help='The timeframe to download') parser.add_argument('--debug', action ='store_true', help=('Print Sizer Debugs')) return parser.parse_args() # Get our arguments args = parse_args() # Get our Exchange try: exchange = getattr (ccxt, args.exchange) () except AttributeError: print('-'*36,' ERROR ','-'*35) print('Exchange "{}" not found. Please check the exchange is supported.'.format(args.exchange)) print('-'*80) quit() # Check if fetching of OHLC Data is supported if exchange.has["fetchOHLCV"] != True: print('-'*36,' ERROR ','-'*35) print('{} does not support fetching OHLC data. Please use another exchange'.format(args.exchange)) print('-'*80) quit() # Check requested timeframe is available. If not return a helpful error. if (not hasattr(exchange, 'timeframes')) or (args.timeframe not in exchange.timeframes): print('-'*36,' ERROR ','-'*35) print('The requested timeframe ({}) is not available from {}\n'.format(args.timeframe,args.exchange)) print('Available timeframes are:') for key in exchange.timeframes.keys(): print(' - ' + key) print('-'*80) quit() # Check if the symbol is available on the Exchange exchange.load_markets() if args.symbol not in exchange.symbols: print('-'*36,' ERROR ','-'*35) print('The requested symbol ({}) is not available from {}\n'.format(args.symbol,args.exchange)) print('Available symbols are:') for key in exchange.symbols: print(' - ' + key) print('-'*80) quit() # Get data since = getUNIX('2017-01-01') data = exchange.fetch_ohlcv(args.symbol, args.timeframe, since) header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume'] df = pd.DataFrame(data, columns=header).set_index('Timestamp') # Save it symbol_out = args.symbol.replace("/","") filename = '{}-{}-{}.csv'.format(args.exchange, symbol_out,args.timeframe) df.to_csv(filename)
Thanks
I was able to solve this one, here is the code for anyone who is interested in.
import ccxt from datetime import datetime, timedelta, timezone import math import argparse import pandas as pd import time import os import sys def parse_args(): parser = argparse.ArgumentParser(description='CCXT Market Data Downloader') parser.add_argument('-s','--symbol', type=str, required=True, help='The Symbol of the Instrument/Currency Pair To Download') parser.add_argument('-e','--exchange', type=str, required=True, help='The exchange to download from') parser.add_argument('-t','--timeframe', type=str, default='1d', choices=['1m', '5m','15m', '30m','1h', '2h', '3h', '4h', '6h', '12h', '1d', '1M', '1y'], help='The timeframe to download') parser.add_argument('-d','--fromdate', type=str, default='2017-01-01 00:00:00', help='The start date to download from (yyyy-mm-dd hh:mm:ss)') parser.add_argument('--debug', action ='store_true', help=('Print Sizer Debugs')) return parser.parse_args() # Get our arguments args = parse_args() # Get our Exchange try: exchange = getattr (ccxt, args.exchange) () except AttributeError: print('-'*36,' ERROR ','-'*35) print('Exchange "{}" not found. Please check the exchange is supported.'.format(args.exchange)) print('-'*80) quit() # Check if fetching of OHLC Data is supported if exchange.has["fetchOHLCV"] != True: print('-'*36,' ERROR ','-'*35) print('{} does not support fetching OHLC data. Please use another exchange'.format(args.exchange)) print('-'*80) quit() # Check requested timeframe is available. If not return a helpful error. if (not hasattr(exchange, 'timeframes')) or (args.timeframe not in exchange.timeframes): print('-'*36,' ERROR ','-'*35) print('The requested timeframe ({}) is not available from {}\n'.format(args.timeframe,args.exchange)) print('Available timeframes are:') for key in exchange.timeframes.keys(): print(' - ' + key) print('-'*80) quit() # Check if the symbol is available on the Exchange exchange.load_markets() if args.symbol not in exchange.symbols: print('-'*36,' ERROR ','-'*35) print('The requested symbol ({}) is not available from {}\n'.format(args.symbol,args.exchange)) print('Available symbols are:') for key in exchange.symbols: print(' - ' + key) print('-'*80) quit() # ----------------------------------------------------------------------------- root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.append(root + '/python') # ----------------------------------------------------------------------------- # common constants msec = 1000 minute = 60 * msec hold = 30 timedic = { '1m': minute, '5m': 5*minute, '15m': 15*minute, '30m': 30*minute, '1h': 60*minute, '2h': 2*60*minute, '3h': 3*60*minute, '4h': 4*60*minute, '6h': 6*60*minute, '12h': 12*60*minute, '1d': 24*60*minute, '1M': 30*24*60*minute, '1y': 365*24*60*minute } # ----------------------------------------------------------------------------- exchange.enableRateLimit = True exchange.rateLimit = 10000 # ----------------------------------------------------------------------------- from_timestamp = exchange.parse8601(args.fromdate) # ----------------------------------------------------------------------------- now = exchange.milliseconds() # ----------------------------------------------------------------------------- # Get data data = [] while from_timestamp < now: try: print(exchange.milliseconds(), 'Fetching candles starting from', exchange.iso8601(from_timestamp)) ohlcvs = exchange.fetch_ohlcv('BTC/USD', args.timeframe, from_timestamp) print(exchange.milliseconds(), 'Fetched', len(ohlcvs), 'candles') if len(ohlcvs) > 0: first = ohlcvs[0][0] last = ohlcvs[-1][0] print('First candle epoch', first, exchange.iso8601(first)) print('Last candle epoch', last, exchange.iso8601(last)) from_timestamp = ohlcvs[-1][0] + timedic.get(args.timeframe) # good data += ohlcvs except (ccxt.ExchangeError, ccxt.AuthenticationError, ccxt.ExchangeNotAvailable, ccxt.RequestTimeout) as error: print('Got an error', type(error).__name__, error.args, ', retrying in', hold, 'seconds...') time.sleep(hold) # ------------------------------------------------------------------------------- #create dataframe header = ['Timestamp', 'Open', 'High', 'Low', 'Close', 'Volume'] df = pd.DataFrame(data, columns=header).set_index('Timestamp') # Save it symbol_out = args.symbol.replace("/","") filename = '{}-{}-{}-{}.csv'.format(args.exchange, symbol_out,args.timeframe,args.fromdate[:10]) df.to_csv(filename)
Thanks
Mariano
-
Hello everyone,
I am trying to test @Rodrigo-Brito 's repo on Binance live.
https://github.com/rodrigo-brito/backtrader-binance-botHere is the base code class for strategy.
class StrategyBase(bt.Strategy): def __init__(self): self.order = None self.last_operation = "SELL" self.status = "DISCONNECTED" self.bar_executed = 0 self.buy_price_close = None self.soft_sell = False self.hard_sell = False self.log("Base strategy initialized") def reset_sell_indicators(self): self.soft_sell = False self.hard_sell = False self.buy_price_close = None def notify_data(self, data, status, *args, **kwargs): self.status = data._getstatusname(status) print(self.status) if status == data.LIVE: self.log("LIVE DATA - Ready to trade") def short(self): if self.last_operation == "SELL": return if config.ENV == config.DEVELOPMENT: self.log("Sell ordered: $%.2f" % self.data0.close[0]) return self.sell() cash, value = self.broker.get_wallet_balance(config.COIN_TARGET) amount = value*0.99 self.log("Sell ordered: $%.2f. Amount %.6f %s - $%.2f USDT" % (self.data0.close[0], amount, config.COIN_TARGET, value), True) return self.sell(size=amount) def long(self): if self.last_operation == "BUY": return self.log("Buy ordered: $%.2f" % self.data0.close[0], True) self.buy_price_close = self.data0.close[0] price = self.data0.close[0] if config.ENV == config.DEVELOPMENT: return self.buy() cash, value = self.broker.get_wallet_balance(config.COIN_REFER) amount = (value / price) * 0.99 # Workaround to avoid precision issues self.log("Buy ordered: $%.2f. Amount %.6f %s. Ballance $%.2f USDT" % (self.data0.close[0], amount, config.COIN_TARGET, value), True) return self.buy(size=amount) def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do self.log('ORDER ACCEPTED/SUBMITTED') self.order = order return if order.status in [order.Expired]: self.log('BUY EXPIRED', True) elif order.status in [order.Completed]: if order.isbuy(): self.last_operation = "BUY" self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm), True) if config.ENV == config.PRODUCTION: print(order.__dict__) print( 'BUY EXECUTED, Price: %.8f, Cost: %.8f, Comm %.8f' % (order.executed.price, order.executed.value, order.executed.comm), True) else: # Sell self.last_operation = "SELL" self.reset_sell_indicators() self.log('SELL EXECUTED, Price: %.8f, Cost: %.8f, Comm %.8f' % (order.executed.price, order.executed.value, order.executed.comm), True) print( 'SELL EXECUTED, Price: %.8f, Cost: %.8f, Comm %.8f' % (order.executed.price, order.executed.value, order.executed.comm), True) # Sentinel to None: new orders allowed elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected: Status %s - %s' % (order.Status[order.status], self.last_operation), True) self.order = None
It works for a while then crashes. I couldn't figure out why. Also order.executed.XXX returns zero and gets unsupported operand type for backtrader/position.py
What am i missing?
Here is the output:
ENV = production Starting Portfolio Value: 0.00045425 DELAYED LIVE {'dteos': 737246.9999999999, 'exectype': 0, 'ordtype': 0, 'size': 0.017, 'position': 0, 'p': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffaced6b940>, 'ref': 1, 'status': 4, 'data': <ccxtbt.ccxtfeed.CCXTFeed object at 0x7ffad7e55f28>, 'executed': <backtrader.order.OrderData object at 0x7ffae95f9438>, 'params': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffaced6b940>, 'broker': None, '_active': True, 'ccxt_order': {'price': 0.02231764705882353, 'datetime': '2019-07-26T19:53:19.784Z', 'symbol': 'ETH/BTC', 'lastTradeTimestamp': None, 'amount': 0.017, 'remaining': 0.0, 'average': 0.02231764705882353, 'fee': None, 'side': 'buy', 'type': 'market', 'id': '441732094', 'cost': 0.0003794, 'timestamp': 1564170799784, 'status': 'closed', 'info': {'price': '0.00000000', 'icebergQty': '0.00000000', 'type': 'MARKET', 'cummulativeQuoteQty': '0.00037940', 'orderId': 441732094, 'executedQty': '0.01700000', 'side': 'BUY', 'isWorking': True, 'stopPrice': '0.00000000', 'time': 1564170799784, 'origQty': '0.01700000', 'symbol': 'ETHBTC', 'updateTime': 1564170799784, 'clientOrderId': 'Ma4x8A2WqBudmPuOtdVEPB', 'timeInForce': 'GTC', 'status': 'FILLED'}, 'filled': 0.017, 'trades': None}, 'owner': <live_strategy.bt_bandtrader_v3_trend.Strategy object at 0x7ffad7e55cc0>, 'info': AutoOrderedDict(), 'created': <backtrader.order.OrderData object at 0x7ffad7e77828>, '_limitoffset': 0.0, '_plimit': None, 'comminfo': None, 'triggered': False} BUY EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True {'dteos': 737246.9999999999, 'exectype': 0, 'ordtype': 0, 'size': 0.017, 'position': 0, 'p': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffaced6b940>, 'ref': 1, 'status': 4, 'data': <ccxtbt.ccxtfeed.CCXTFeed object at 0x7ffad7e55f28>, 'executed': <backtrader.order.OrderData object at 0x7ffae95f9438>, 'params': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffaced6b940>, 'broker': None, '_active': True, 'ccxt_order': {'price': 0.02231764705882353, 'datetime': '2019-07-26T19:53:19.784Z', 'symbol': 'ETH/BTC', 'lastTradeTimestamp': None, 'amount': 0.017, 'remaining': 0.0, 'average': 0.02231764705882353, 'fee': None, 'side': 'buy', 'type': 'market', 'id': '441732094', 'cost': 0.0003794, 'timestamp': 1564170799784, 'status': 'closed', 'info': {'price': '0.00000000', 'icebergQty': '0.00000000', 'type': 'MARKET', 'cummulativeQuoteQty': '0.00037940', 'orderId': 441732094, 'executedQty': '0.01700000', 'side': 'BUY', 'isWorking': True, 'stopPrice': '0.00000000', 'time': 1564170799784, 'origQty': '0.01700000', 'symbol': 'ETHBTC', 'updateTime': 1564170799784, 'clientOrderId': 'Ma4x8A2WqBudmPuOtdVEPB', 'timeInForce': 'GTC', 'status': 'FILLED'}, 'filled': 0.017, 'trades': None}, 'owner': <live_strategy.bt_bandtrader_v3_trend.Strategy object at 0x7ffad7e55cc0>, 'info': AutoOrderedDict(), 'created': <backtrader.order.OrderData object at 0x7ffad7e77828>, '_limitoffset': 0.0, '_plimit': None, 'comminfo': None, 'triggered': False} BUY EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True SELL EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True SELL EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True {'dteos': 737246.9999999999, 'exectype': 0, 'ordtype': 0, 'size': 0.127, 'position': 0, 'p': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef8a3c8>, 'ref': 3, 'status': 4, 'data': <ccxtbt.ccxtfeed.CCXTFeed object at 0x7ffad7e55f28>, 'executed': <backtrader.order.OrderData object at 0x7ffacef46518>, 'params': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef8a3c8>, 'broker': None, '_active': True, 'ccxt_order': {'price': 0.022296929133858265, 'datetime': '2019-07-26T19:55:23.200Z', 'symbol': 'ETH/BTC', 'lastTradeTimestamp': None, 'amount': 0.127, 'remaining': 0.0, 'average': 0.022296929133858265, 'fee': None, 'side': 'buy', 'type': 'market', 'id': '441732983', 'cost': 0.00283171, 'timestamp': 1564170923200, 'status': 'closed', 'info': {'price': '0.00000000', 'icebergQty': '0.00000000', 'type': 'MARKET', 'cummulativeQuoteQty': '0.00283171', 'orderId': 441732983, 'executedQty': '0.12700000', 'side': 'BUY', 'isWorking': True, 'stopPrice': '0.00000000', 'time': 1564170923200, 'origQty': '0.12700000', 'symbol': 'ETHBTC', 'updateTime': 1564170923200, 'clientOrderId': '2ZWJFmhksnBVwpql2Ur0C4', 'timeInForce': 'GTC', 'status': 'FILLED'}, 'filled': 0.127, 'trades': None}, 'owner': <live_strategy.bt_bandtrader_v3_trend.Strategy object at 0x7ffad7e55cc0>, 'info': AutoOrderedDict(), 'created': <backtrader.order.OrderData object at 0x7ffacef465f8>, '_limitoffset': 0.0, '_plimit': None, 'comminfo': None, 'triggered': False} BUY EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True {'dteos': 737246.9999999999, 'exectype': 0, 'ordtype': 0, 'size': 0.127, 'position': 0, 'p': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef8a3c8>, 'ref': 3, 'status': 4, 'data': <ccxtbt.ccxtfeed.CCXTFeed object at 0x7ffad7e55f28>, 'executed': <backtrader.order.OrderData object at 0x7ffacef46518>, 'params': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef8a3c8>, 'broker': None, '_active': True, 'ccxt_order': {'price': 0.022296929133858265, 'datetime': '2019-07-26T19:55:23.200Z', 'symbol': 'ETH/BTC', 'lastTradeTimestamp': None, 'amount': 0.127, 'remaining': 0.0, 'average': 0.022296929133858265, 'fee': None, 'side': 'buy', 'type': 'market', 'id': '441732983', 'cost': 0.00283171, 'timestamp': 1564170923200, 'status': 'closed', 'info': {'price': '0.00000000', 'icebergQty': '0.00000000', 'type': 'MARKET', 'cummulativeQuoteQty': '0.00283171', 'orderId': 441732983, 'executedQty': '0.12700000', 'side': 'BUY', 'isWorking': True, 'stopPrice': '0.00000000', 'time': 1564170923200, 'origQty': '0.12700000', 'symbol': 'ETHBTC', 'updateTime': 1564170923200, 'clientOrderId': '2ZWJFmhksnBVwpql2Ur0C4', 'timeInForce': 'GTC', 'status': 'FILLED'}, 'filled': 0.127, 'trades': None}, 'owner': <live_strategy.bt_bandtrader_v3_trend.Strategy object at 0x7ffad7e55cc0>, 'info': AutoOrderedDict(), 'created': <backtrader.order.OrderData object at 0x7ffacef465f8>, '_limitoffset': 0.0, '_plimit': None, 'comminfo': None, 'triggered': False} BUY EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True SELL EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True SELL EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True {'dteos': 737246.9999999999, 'exectype': 0, 'ordtype': 0, 'size': 0.126, 'position': 0, 'p': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef46d68>, 'ref': 5, 'status': 4, 'data': <ccxtbt.ccxtfeed.CCXTFeed object at 0x7ffad7e55f28>, 'executed': <backtrader.order.OrderData object at 0x7ffacef464a8>, 'params': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef46d68>, 'broker': None, '_active': True, 'ccxt_order': {'price': 0.022314285714285714, 'datetime': '2019-07-26T19:56:20.376Z', 'symbol': 'ETH/BTC', 'lastTradeTimestamp': None, 'amount': 0.126, 'remaining': 0.0, 'average': 0.022314285714285714, 'fee': None, 'side': 'buy', 'type': 'market', 'id': '441733466', 'cost': 0.0028116, 'timestamp': 1564170980376, 'status': 'closed', 'info': {'price': '0.00000000', 'icebergQty': '0.00000000', 'type': 'MARKET', 'cummulativeQuoteQty': '0.00281160', 'orderId': 441733466, 'executedQty': '0.12600000', 'side': 'BUY', 'isWorking': True, 'stopPrice': '0.00000000', 'time': 1564170980376, 'origQty': '0.12600000', 'symbol': 'ETHBTC', 'updateTime': 1564170980376, 'clientOrderId': 'TwXVKXWuXd1MRZ7Fcmjruk', 'timeInForce': 'GTC', 'status': 'FILLED'}, 'filled': 0.126, 'trades': None}, 'owner': <live_strategy.bt_bandtrader_v3_trend.Strategy object at 0x7ffad7e55cc0>, 'info': AutoOrderedDict(), 'created': <backtrader.order.OrderData object at 0x7ffacef46eb8>, '_limitoffset': 0.0, '_plimit': None, 'comminfo': None, 'triggered': False} BUY EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True {'dteos': 737246.9999999999, 'exectype': 0, 'ordtype': 0, 'size': 0.126, 'position': 0, 'p': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef46d68>, 'ref': 5, 'status': 4, 'data': <ccxtbt.ccxtfeed.CCXTFeed object at 0x7ffad7e55f28>, 'executed': <backtrader.order.OrderData object at 0x7ffacef464a8>, 'params': <backtrader.metabase.AutoInfoClass_OrderBase_CCXTOrder object at 0x7ffacef46d68>, 'broker': None, '_active': True, 'ccxt_order': {'price': 0.022314285714285714, 'datetime': '2019-07-26T19:56:20.376Z', 'symbol': 'ETH/BTC', 'lastTradeTimestamp': None, 'amount': 0.126, 'remaining': 0.0, 'average': 0.022314285714285714, 'fee': None, 'side': 'buy', 'type': 'market', 'id': '441733466', 'cost': 0.0028116, 'timestamp': 1564170980376, 'status': 'closed', 'info': {'price': '0.00000000', 'icebergQty': '0.00000000', 'type': 'MARKET', 'cummulativeQuoteQty': '0.00281160', 'orderId': 441733466, 'executedQty': '0.12600000', 'side': 'BUY', 'isWorking': True, 'stopPrice': '0.00000000', 'time': 1564170980376, 'origQty': '0.12600000', 'symbol': 'ETHBTC', 'updateTime': 1564170980376, 'clientOrderId': 'TwXVKXWuXd1MRZ7Fcmjruk', 'timeInForce': 'GTC', 'status': 'FILLED'}, 'filled': 0.126, 'trades': None}, 'owner': <live_strategy.bt_bandtrader_v3_trend.Strategy object at 0x7ffad7e55cc0>, 'info': AutoOrderedDict(), 'created': <backtrader.order.OrderData object at 0x7ffacef46eb8>, '_limitoffset': 0.0, '_plimit': None, 'comminfo': None, 'triggered': False} BUY EXECUTED, Price: 0.00000000, Cost: 0.00000000, Comm 0.00000000 True Finished with error: unsupported operand type(s) for *: 'NoneType' and 'float' Traceback (most recent call last): File "/home/tunc/PycharmProjects/backtesting/main.py", line 109, in <module> except KeyboardInterrupt: File "/home/tunc/PycharmProjects/backtesting/main.py", line 94, in main File "/home/tunc/PycharmProjects/backtesting/venv/lib/python3.5/site-packages/backtrader/cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "/home/tunc/PycharmProjects/backtesting/venv/lib/python3.5/site-packages/backtrader/cerebro.py", line 1298, in runstrategies self._runnext(runstrats) File "/home/tunc/PycharmProjects/backtesting/venv/lib/python3.5/site-packages/backtrader/cerebro.py", line 1623, in _runnext self._brokernotify() File "/home/tunc/PycharmProjects/backtesting/venv/lib/python3.5/site-packages/backtrader/cerebro.py", line 1360, in _brokernotify self._broker.next() File "/home/tunc/PycharmProjects/backtesting/venv/lib/python3.5/site-packages/ccxtbt/ccxtbroker.py", line 203, in next pos.update(o_order.size, o_order.price) File "/home/tunc/PycharmProjects/backtesting/venv/lib/python3.5/site-packages/backtrader/position.py", line 193, in update self.price = (self.price * oldsize + size * price) / self.size TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
-
@mr-kite I'm running into the same problem. I think it's when you have two separate orders and backtrader is trying to compute your average order price. After you've completed an order, if you print out
self.broker.getposition(self.data)
then you'll see that 'Size' will have a value, but 'Price' has 'None' for its value and it can't handle that. So the question is how can we fill in the 'Price' variable with the correct value? -
@backtrader @Ed-Bartosh Do you have any insight you're willing to share for how mr-kite and I might fix our problem? I've been trying to dig into it for the last couple of days as a programming newb, and it's over my head. It has something to do with how data is passed around I think, but I'm lost as heck.
-
Hi everyone.
About the problem with the position calculation in my simple example. I think it is a problem with bt-ccxt-store. I create a workarround, ignoring the position value and store this value manually inside the Strategy Class. Other customization is thebuy()
andsell()
functions, the sizer does not work very well with binance. When the fee is charged in BNB, it does not include the wallet value. I'm using this class to create custom buy and sell operations. For now, I'm using backtrader to get the historical information and indictators. All other operations I'm doing manually.
Custom code: https://gist.github.com/rodrigo-brito/8c82020f04e946e3f0c39c7243cfe1ee -
@Rodrigo-Brito I think I may have figured out a possible solution. In ccxtbroker.py after line 219 where it says
order = CCXTOrder(owner, data, _order)
add another line to update 'order' like thisorder.price = ret_ord['price']
Then the '_submit' function will be returning price data. I'm not sure that this is the best fix, but it seems to be working for me for now.Original function
def _submit(self, owner, data, exectype, side, amount, price, params): order_type = self.order_types.get(exectype) if exectype else 'market' # Extract CCXT specific params if passed to the order params = params['params'] if 'params' in params else params ret_ord = self.store.create_order(symbol=data.symbol, order_type=order_type, side=side, amount=amount, price=price, params=params) _order = self.store.fetch_order(ret_ord['id'], data.symbol) order = CCXTOrder(owner, data, _order) self.open_orders.append(order) self.notify(order) return order
Modified function
def _submit(self, owner, data, exectype, side, amount, price, params): order_type = self.order_types.get(exectype) if exectype else 'market' # Extract CCXT specific params if passed to the order params = params['params'] if 'params' in params else params ret_ord = self.store.create_order(symbol=data.symbol, order_type=order_type, side=side, amount=amount, price=price, params=params) _order = self.store.fetch_order(ret_ord['id'], data.symbol) order = CCXTOrder(owner, data, _order) order.price = ret_ord['price'] self.open_orders.append(order) self.notify(order) return order
-
@ThisMustBeTrue Thanks for your debugging. It works flawlessly. But for better and true logging, user must call order data manually in "notify_order()" function. This is my custom function for multiple coins either for backtesting and live production. I added these lines:
def notify_order(self, order, *args, **kwargs): dict = order.__dict__ target = self.data._name.replace('/USDT', '') if ENV == PRODUCTION: exec_time = dict['ccxt_order']['datetime'] exec_price = dict['ccxt_order']['average'] pair = dict['ccxt_order']['symbol'] cost = dict['ccxt_order']['cost'] size = dict['ccxt_order']['filled'] target = pair.replace('/USDT', '') else: pass ....#continues as same.
-
@Rodrigo-Brito Hello Rodrigo, thank you very much for your cool Trading Bot, it really gives me a great learning opportunity.
-
MarketLab offers data recorded in realtime, open source libraries and documentation to backtest easly your algorithms.
Have a look on our website and get started for free: https://www.market-lab.app/Regards,
Charles