Anyone use backtrader to do live trading on Bitcoin exchange?

[Ed Bartosh]

@samk said in Anyone use backtrader to do live trading on Bitcoin exchange?:

@Ed-Bartosh

Thank you I found it. Instead of cloning github, which obviously caused me many issues, I have went ahead and saved the files from your github manually. These files being 'ccxtbroker.py' (which holds the class CCXTBroker) and 'ccxt.py' (which holds the class CCXT ).

You don't need clone anything or download anything manually. All you need is to use pip:

pip install git+https://github.com/bartosh/backtrader.git@ccxt

It worked for Søren Pallesen and should work for you as your setup(anaconda) is similar to yours.

I am receiving an error however saying:

***Traceback (most recent call last):

Hm, that's strange. I'd attribute it to semi-manuall installation you're doing.
Can you run this command and show the output here?

python -c "import ccxt; print(getattr(ccxt, 'gemini'))"

Here is what I'm getting on my machine:

(backtrader-ccxt) backtrader-ccxt (ccxt) $ python -c "import ccxt; print(getattr(ccxt, 'gemini'))"
<class 'ccxt.gemini.gemini'>

[kazi]

Hi guys,

Thank you for the great work being done! I have tested out the bt-ccxt-test.py using hitbtc2 and it works well, the only issue being http timeouts on both the ticker and balance functions eg. 502 etc errors. I would like to know if there is away to handle these errors "gracefully" eg.

while True:
    try:
          async for ticker in poll():
                 # Do work....
    except Exception as e:
            print("error occured, Total errors: ", count)
            time.sleep(5)
            print("Resuming")
            pass

Thank you in advance!

[Ed Bartosh]

@kazi said in Anyone use backtrader to do live trading on Bitcoin exchange?:

Hi guys,

Thank you for the great work being done! I have tested out the bt-ccxt-test.py using hitbtc2 and it works well, the only issue being http timeouts on both the ticker and balance functions eg. 502 etc errors. I would like to know if there is away to handle these errors "gracefully" eg.

Can you show more details(full traceback) on how does it fail?

[kazi]

Hi @Ed-Bartosh ,

Thanks for the quick response, here is an example:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 324, in fetch
    response = opener.open(request, timeout=int(self.timeout / 1000))
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1321, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bt-ccxt-test.py", line 177, in <module>
    sys.exit(runstrategy(sys.argv))
  File "bt-ccxt-test.py", line 174, in runstrategy
    cerebro.run()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1295, in runstrategies
    self._runnext(runstrats)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1626, in _runnext
    strat._next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 329, in _next
    self._next_observers(minperstatus)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 357, in _next_observers
    observer._next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/lineiterator.py", line 275, in _next
    self.next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/observers/broker.py", line 110, in next
    self.lines.value[0] = value = self._owner.broker.getvalue()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/brokers/ccxtbroker.py", line 71, in getvalue
    return self.exchange.fetch_balance()['total'][self.currency]
  File "/usr/local/lib/python3.6/dist-packages/ccxt/hitbtc2.py", line 636, in fetch_balance
    balances = getattr(self, method)()
  File "/usr/local/lib/python3.6/dist-packages/ccxt/hitbtc2.py", line 1001, in request
    response = self.fetch2(path, api, method, params, headers, body)
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 282, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 330, in fetch
    raise RequestTimeout(' '.join([self.id, method, url, 'request timeout']))
ccxt.base.errors.RequestTimeout: hitbtc2 GET https://api.hitbtc.com/api/2/trading/balance request timeout
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 324, in fetch
    response = opener.open(request, timeout=int(self.timeout / 1000))
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1321, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bt-ccxt-test.py", line 177, in <module>
    sys.exit(runstrategy(sys.argv))
  File "bt-ccxt-test.py", line 174, in runstrategy
    cerebro.run()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1295, in runstrategies
    self._runnext(runstrats)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1626, in _runnext
    strat._next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 329, in _next
    self._next_observers(minperstatus)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 357, in _next_observers
    observer._next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/lineiterator.py", line 275, in _next
    self.next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/observers/broker.py", line 110, in next
    self.lines.value[0] = value = self._owner.broker.getvalue()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/brokers/ccxtbroker.py", line 71, in getvalue
    return self.exchange.fetch_balance()['total'][self.currency]
  File "/usr/local/lib/python3.6/dist-packages/ccxt/hitbtc2.py", line 636, in fetch_balance
    balances = getattr(self, method)()
  File "/usr/local/lib/python3.6/dist-packages/ccxt/hitbtc2.py", line 1001, in request
    response = self.fetch2(path, api, method, params, headers, body)
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 282, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 330, in fetch
    raise RequestTimeout(' '.join([self.id, method, url, 'request timeout']))
ccxt.base.errors.RequestTimeout: hitbtc2 GET https://api.hitbtc.com/api/2/trading/balance request timeout

[Mikk Laos]

@Ed-Bartosh

cerebro = bt.Cerebro()

    my_symbol = "BTC_USD"
    exchange = "poloniex"
    # Create broker
    broker_config = {
        'apiKey': '####',
        'secret': '#######',
        'nonce': lambda: str(int(time.time() * 1000))
    }
    broker = bt.brokers.CCXTBroker(exchange=exchange, currency='USDT', config=broker_config)
    cerebro.setbroker(broker)

    my_symbol = "BTC/USDT"

    # Create data feeds
    data_ticks = bt.feeds.CCXT(exchange=exchange, symbol=my_symbol, name="getmarketsummary",
                               timeframe=bt.TimeFrame.Ticks, compression=1)
    cerebro.adddata(data_ticks)

    hist_start_date = datetime.utcnow() - timedelta(minutes=30)
    data_min = bt.feeds.CCXT(exchange=exchange, symbol=my_symbol, name="btc_usd_min",
                             timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date)

   cerebro.adddata(data_min)

    # Add the strategy
    cerebro.addstrategy(TestStrategy)

    # Run the strategy
    cerebro.run()

[Ed Bartosh]

@Mikk-Laos said in Anyone use backtrader to do live trading on Bitcoin exchange?:

data_min = bt.feeds.CCXT(exchange=exchange, symbol=my_symbol, name="btc_usd_min",
                         timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date)

This feed should request data in 1 minute time frame. If exchange doesn't support this it should fail. I'm surprised it worked after your removed granularities < 5m.
Can you show me the output of the test script without your changes?

[Ed Bartosh]

@kazi said in Anyone use backtrader to do live trading on Bitcoin exchange?:

Traceback (most recent call last):
  File "bt-ccxt-test.py", line 177, in <module>
    sys.exit(runstrategy(sys.argv))
  File "bt-ccxt-test.py", line 174, in runstrategy
    cerebro.run()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1295, in runstrategies
    self._runnext(runstrats)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/cerebro.py", line 1626, in _runnext
    strat._next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 329, in _next
    self._next_observers(minperstatus)
  File "/usr/local/lib/python3.6/dist-packages/backtrader/strategy.py", line 357, in _next_observers
    observer._next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/lineiterator.py", line 275, in _next
    self.next()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/observers/broker.py", line 110, in next
    self.lines.value[0] = value = self._owner.broker.getvalue()
  File "/usr/local/lib/python3.6/dist-packages/backtrader/brokers/ccxtbroker.py", line 71, in getvalue
    return self.exchange.fetch_balance()['total'][self.currency]
  File "/usr/local/lib/python3.6/dist-packages/ccxt/hitbtc2.py", line 636, in fetch_balance
    balances = getattr(self, method)()
  File "/usr/local/lib/python3.6/dist-packages/ccxt/hitbtc2.py", line 1001, in request
    response = self.fetch2(path, api, method, params, headers, body)
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 282, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/usr/local/lib/python3.6/dist-packages/ccxt/base/exchange.py", line 330, in fetch
    raise RequestTimeout(' '.join([self.id, method, url, 'request timeout']))
ccxt.base.errors.RequestTimeout: hitbtc2 GET https://api.hitbtc.com/api/2/trading/balance request timeout

Interesting. Is it some kind of networking issue? Are you behind the proxy?

I'll try to implement repeating the request sevaral times if this kind of error occurs. Thank you for reporting!

[samk]

@Ed-Bartosh

So I went ahead again and gave a crack at it trying to install using:

pip install git+https://github.com/bartosh/backtrader.git@ccxt

Here is what went into command line:

C:\Users\Sam>pip install git+https://github.com/bartosh/backtrader.git@ccxt
Collecting git+https://github.com/bartosh/backtrader.git@ccxt
Cloning https://github.com/bartosh/backtrader.git (to ccxt) to c:\users\sam\appdata\local\temp\pip-merrrk-build
Installing collected packages: backtrader
Running setup.py install for backtrader ... done
Successfully installed backtrader-1.9.59.122

Now, my question is, looks like it was cloned to the directory:

c:\users\sam\appdata\local\temp\pip-merrrk-build

However, I work my python environment in:

c:\users\sam\anaconda64best\

Am I having an issue Ed because I need to copy everything from the first directory to my Python working directory? Then it should work?

Thanks for the help.........

[Ed Bartosh]

@samk said in Anyone use backtrader to do live trading on Bitcoin exchange?:

@Ed-Bartosh

So I went ahead again and gave a crack at it trying to install using:

pip install git+https://github.com/bartosh/backtrader.git@ccxt

Here is what went into command line:

C:\Users\Sam>pip install git+https://github.com/bartosh/backtrader.git@ccxt
Collecting git+https://github.com/bartosh/backtrader.git@ccxt
Cloning https://github.com/bartosh/backtrader.git (to ccxt) to c:\users\sam\appdata\local\temp\pip-merrrk-build
Installing collected packages: backtrader
Running setup.py install for backtrader ... done
Successfully installed backtrader-1.9.59.122

Now, my question is, looks like it was cloned to the directory:

c:\users\sam\appdata\local\temp\pip-merrrk-build

Yes, it cloned git repository to that temporary directory, run setup.py and remove that directory. This should install it into Python system path.

However, I work my python environment in:

c:\users\sam\anaconda64best\

That doesn't matter. Any python module that is installed with pip should be available from any directory.

Am I having an issue Ed because I need to copy everything from the first directory to my Python working directory? Then it should work?

you don't need to copy anything anywhere. It should just work after the installation.

Can you run this from the command line, please?

python -c "import ccxt; print(getattr(ccxt, 'gemini'))"

[kazi]

@Ed-Bartosh I am running the script on a vps on the west side of the USA and I am not going through any proxies. I have relatively good connectivity. I am trying to figure out if there is a reliable way to handle connection errors. The stability of the exchange is also hard to predict. Thanks for having a look!

[Søren Pallesen]

@Ed-Bartosh I see you are potentially merging a PR for "Added config to CCXT feed parameters" - I also need to run both against both private and public sandbox APIs for both Feeds and Broker.

Im a bit confused how (and from where) should a pass the config params and in what format?

Thx :-)

[Ed Bartosh]

@Søren-Pallesen I didn't merge that PR yet. I will when it's fixed.

I think the config for the feed should be in the same format as for the broker:

config = {'urls': {'api': 'https://api.sandbox.gemini.com'},
                     'apiKey': 'XXXXX',
                     'secret': 'XXXXX',
                     'nonce': lambda: str(int(time.time() * 1000))
                    }
broker = bt.brokers.CCXTBroker(exchange='gemini',
                               currency='USD', config=config)
cerebro.setbroker(broker)

# Create data feeds
data_ticks = bt.feeds.CCXT(exchange='geminy', symbol='BTC/USD',
                          name="btc_usd_tick",
                          timeframe=bt.TimeFrame.Ticks,
                          compression=1, config=config)
cerebro.adddata(data_ticks)

[Søren Pallesen]

@Ed-Bartosh Thx - agree.

Follow-up question: Is it possible to pass config params to an order? I am thinking about this in the context of live trading where individual exchanges support various order types and params - Trailing Stop is a good example, its implemented in may different ways.

I prefer to use the native order types on the exchange rather than calculating them custom in the Next loop.

[Ed Bartosh]

@Søren-Pallesen said in Anyone use backtrader to do live trading on Bitcoin exchange?:

@Ed-Bartosh Thx - agree.

Follow-up question: Is it possible to pass config params to an order? I am thinking about this in the context of live trading where individual exchanges support various order types and params - Trailing Stop is a good example, its implemented in may different ways.

Isn't it supported in backtrader? https://www.backtrader.com/docu/order-creation-execution/trail/stoptrail.html

I prefer to use the native order types on the exchange rather than calculating them custom in the Next loop.

Then it's probably better for your to use native brokerage API instead of backtrader and ccxt. Both backtrader and ccxt generalize brokerage APIs in order to be able to easily switch between brokers without changong the strategy code. Unfortunately this means that some brokerage-specific functions are not available.

[Søren Pallesen]

@Ed-Bartosh I only want to use the native CCXT capabilities, as custom params are support: https://github.com/ccxt/ccxt/wiki/Manual#custom-order-params

[Ed Bartosh]

@Søren-Pallesen said in Anyone use backtrader to do live trading on Bitcoin exchange?:

@Ed-Bartosh I only want to use the native CCXT capabilities, as custom params are support: https://github.com/ccxt/ccxt/wiki/Manual#custom-order-params

This should be possible even with current code. I didn't test it, but you can try something like this:

order = self.buy(data, exectype=bt.Order.Limit, size=10, price=data.close[0], kwargs={'trading_agreement': 'agree'})

kwargs are passed to ccxt create_order API as 'params' parameter.

Please, make sure that your strategy may break when you switch to another broker if you use this approach.

[Søren Pallesen]

@Ed-Bartosh ahhh excellent - thx again again ... will try ...yes will off cause use this with caution :-)

[samk]

@Ed-Bartosh

I did that. This is what I get:

C:\Users\Sam>python -c "import ccxt; print(getattr(ccxt, 'gemini'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named ccxt

Keep in mind, this is after this was completed too:

C:\Users\Sam>pip install git+https://github.com/bartosh/backtrader.git@ccxt
Collecting git+https://github.com/bartosh/backtrader.git@ccxt
Cloning https://github.com/bartosh/backtrader.git (to ccxt) to c:\users\sam\appdata\local\temp\pip-merrrk-build
Installing collected packages: backtrader
Running setup.py install for backtrader ... done
Successfully installed backtrader-1.9.59.122

Which is very odd because I have already pip installed and imported ccxt.......

[Ed Bartosh]

@samk said in Anyone use backtrader to do live trading on Bitcoin exchange?:

@Ed-Bartosh

I did that. This is what I get:

C:\Users\Sam>python -c "import ccxt; print(getattr(ccxt, 'gemini'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named ccxt

Keep in mind, this is after this was completed too:

C:\Users\Sam>pip install git+https://github.com/bartosh/backtrader.git@ccxt
Collecting git+https://github.com/bartosh/backtrader.git@ccxt
Cloning https://github.com/bartosh/backtrader.git (to ccxt) to c:\users\sam\appdata\local\temp\pip-merrrk-build
Installing collected packages: backtrader
Running setup.py install for backtrader ... done
Successfully installed backtrader-1.9.59.122

Which is very odd because I have already pip installed and imported ccxt.......

It's odd indeed! My guess is that pip is configured to use different version of python. Can you run this, please:

python --version
python3 --version

[Ed Bartosh]

@Søren-Pallesen said in Anyone use backtrader to do live trading on Bitcoin exchange?:

@Ed-Bartosh ahhh excellent - thx again again ... will try ...yes will off cause use this with caution :-)

Small correction. Correct call should probably look like this:

order = self.buy(data, exectype=bt.Order.Limit, size=10,
                             price=data.close[0],
                            trading_agreement='agree')