Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    1. Home
    2. kriku
    3. Posts
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/
    K
    • Profile
    • Following 0
    • Followers 0
    • Topics 4
    • Posts 20
    • Best 0
    • Groups 0

    Posts made by kriku

    • RE: Sizer trouble

      Relevant part from the documentation:

      And although people can try to look into the future with a positive [1] index approach, this requires preloading data which is not always available.
      

      https://www.backtrader.com/docu/cerebro/cheat-on-open/cheat-on-open/

      posted in General Code/Help
      K
      kriku
    • RE: Sizer trouble

      @run-out Btw, the exact method to determine the size of the order is quite irrelevant, be it with data.open[1] or not. The log shows that a MARGIN status is raised although the order + commission is less than available cash.

      posted in General Code/Help
      K
      kriku
    • RE: Sizer trouble

      @run-out That's not true. There are a number of posts on the issue, for example this: https://community.backtrader.com/topic/260/how-to-buy-maximum-stakes

      My code clearly demonstrates that data.open[1] retrieves the next open price.

      posted in General Code/Help
      K
      kriku
    • RE: Sizer trouble

      BTW, I have removed the $50 buffer from the Sizer so that the line

      if result * data.open[1] + c < self.broker.get_cash()-50:
      

      is now as following:

      if result * data.open[1] + c < self.broker.get_cash():
      
      posted in General Code/Help
      K
      kriku
    • RE: Sizer trouble

      Sorry for the delay.

      @run-out said in Sizer trouble:

      1. What is this variable that's showing up? data.open[1]

      data.open[1] is the next bar of the data feed (the nearest bar in the future) as data.open[0] is the current bar. As you see, the open price value by which the sizing is done (10.685), is the same as the open price value printed out by the order status change handler in the MARGIN message. For the sizer, it is open[1] and for the order status change handler it is open[0] because the handler is triggered at the next bar.

      1. Could you print out logs with labels so we can follow along without hunting? Sizer: 10.814 9347 46.735 101125.193 101180.15400000001
      print("Sizer:", data.open[1], result, c, result * data.open[1] + c, self.broker.get_cash())
      

      Prints out the following:

      • the next open bar value used for sizing;
      • result of the _get_sizing (the size determined in the current function);
      • commission as c;
      • size * open price of the next bar + commission (by my current understanding, the total cost of the transaction);
      • amount of cash obtained from the broker.
      1. Could you print out a log where margin is happening with OHLCV, both sides of the trade please, along with the margin error message?

      Here you are:

      dt = self.datas[0].datetime.datetime()
      print(dt, order.p.data.open[0], order.p.data.high[0], order.p.data.low[0], order.p.data.close[0], order.p.data.volume[0])
      

      2005-03-23 11:00:00 10.668 10.728 10.625 10.687 9830985.0
      Sizer: 10.685 10009 50.045 106996.21 106998.12
      2005-03-23 12:00:00 MARGIN! 10.685 10009 106998.12
      2005-03-23 12:00:00 10.685 10.771 10.685 10.719 7653056.0

      Looks like the broker is trying to add some additional costs besides open price & commission but I have no idea what these could possibly be.

      posted in General Code/Help
      K
      kriku
    • RE: Sizer trouble

      I ended up by disabling the check as suggested here: https://community.backtrader.com/topic/782/help-with-simple-bband-strategy

      cerebro.broker.set_checksubmit(checksubmit=False)
      

      However, I would still appreciate to learn what's wrong with the Sizer or how to debug the issue further. Looks like a BT bug for me :S

      posted in General Code/Help
      K
      kriku
    • Sizer trouble

      Some (not all!) of my orders get MARGIN rejected and I cannot figure out why. Please help, thanks in advance.

      Below I buffer the orders by 50 USD in the Sizer but I still get the MARGIN :(

      _getsizing:

      result = int(cash_per_order / data.open[1]) * self.coef
      result = round(result, 0)
      while True:
              c = comminfo.getcommission(result, data.open[1])
              if result * data.open[1] + c < self.broker.get_cash()-50:
                      break
              result -= 1
      
      # price, size, commission, all costs, available cash
      print("Sizer:", data.open[1], result, c, result * data.open[1] + c, self.broker.get_cash()) 
      

      notify_order:

      elif (o_status == "Margin"):
              self.cancel(self.order[symbol])
              self.order[symbol] = None
              print("\n", "MARGIN!", order.p.data.open[0], order.size, self.broker.get_cash(), "\n")
      

      Result:
      Sizer: 10.814 9347 46.735 101125.193 101180.15400000001

      MARGIN! 10.814 9347 101180.15400000001

      The whole (very straightforward) Sizer:

      class LongSizer(bt.Sizer):
      
          #coef = 0.95
          coef = 1
      
          def _getsizing(self, comminfo, cash, data, isbuy):
      
              result = 0
      
              if isbuy is True:
      
                  if self.strategy.getposition(data) is not None:
                      if self.strategy.getposition(data).size > 0:
                          return 0
      
                  max_pos_value = int(self.broker.get_value() / self.strategy.max_stake)
      
                  cash_per_order = int(self.broker.get_cash() / len(self.strategy.buy_orders.keys()))
                  if cash_per_order > max_pos_value:
                      cash_per_order = max_pos_value
      
                  try:
                      if  cash_per_order > data.open[1]:
                          result = int(cash_per_order / data.open[1]) * self.coef
                          result = round(result, 0)
                          while True:
                              c = comminfo.getcommission(result, data.open[1])
                              if result * data.open[1] + c < self.broker.get_cash()-50:
                                  break
                              result -= 1
      
                          print("Sizer:", data.open[1], result, c, result * data.open[1] + c, self.broker.get_cash())
      
                      else:
                          return 0
                  except IndexError:
                      return 0
      
              else:
                  if self.strategy.getposition(data) is not None:
                      if self.strategy.getposition(data).size > 0:
                          result = self.strategy.getposition(data).size
      
              return result
      
      posted in General Code/Help
      K
      kriku
    • Using buffered indicator data - broker.get_value() 'nan'

      I thought that the following could be of interest for someone.

      I try to develop a swing trading strategy. By definition, it is all about good entry and exit points. For every individual stock, there could be any number of entry points but scanning hundreds of them results in a reasonably stable flow.

      Entry point rules are based on both indicators of the single stock and the whole bunch of them. For instance, the strategy does not enter a trade if the average price of the sector is falling.

      As hundreds of lines are scanned and only 2-3 positions maintained simultaneously, it is not known how many trades could be made with any single stock. However, I would like to know it. Cutting the list down to the stock in question is not a solution because it would disable the criteria basing on the average price movements of the whole list.

      I tried to be smart and cached the sector data on disk and loaded it as separate lines:

      class BufferedSectorIndicatorsCSVData(bt.CSVDataBase):
      
          lines = ('datetime', 'sector_mean', 'hull300_slope', 'sma_slope',)
      
          def _loadline(self, linetokens):
              i = itertools.count(0)
              date_txt = linetokens[next(i)]
              time_txt = linetokens[next(i)]
      
              dtxt = date_txt + " " + time_txt
      
              y = int(dtxt[0:4])
              m = int(dtxt[5:7])
              d = int(dtxt[8:10])
              h = int(dtxt[11:13])
              minute = int(dtxt[14:16])
      
              dt = datetime(year=y, month=m, day=d, hour=h, minute=minute)
      
              dtnum = bt.date2num(dt)
      
              self.lines.datetime[0] = dtnum
              self.lines.sector_mean[0] = float(linetokens[next(i)])
              self.lines.hull300_slope[0] = float(linetokens[next(i)])
              self.lines.sma_slope[0] = float(linetokens[next(i)])
      
              return True
      

      However, this fails in quite an interesting way. Everything works but broker.get_value() starts to return 'nan' even for the stock that has correct price data. It can be observed in different ways; I checked it in the Sizer with the debugger and saw that all data.tick_* fields were correctly filled in and the data.p.dataname pointed to the real stock data (that I was not accidentally trying to "buy" a cached indicator data). Therefore the issue is not exactly similar to the one discussed here: https://community.backtrader.com/topic/1373/why-cerebro-broker-getvalue-is-nan

      sizer_value_nan.png

      The solution was to add fictive OLCH data to the buffered lines:

              self.lines.open[0] = 0
              self.lines.close[0] = 0
              self.lines.high[0] = 0
              self.lines.low[0] = 0
              self.lines.volume[0] = 0
              self.lines.openinterest[0] = 0
      

      Is this a bug or a feature?

      Are there better / more backtraderish ways to use cached data?

      posted in Indicators/Strategies/Analyzers
      K
      kriku
    • RE: Maximum number of data lines?

      Right now about 1/2 of the execution time is spent on the indicators and 1/8 on trading (next() and other things). I cannot see how faster data loading could influence that - although I agree with you that I need to load data more efficiently.

      posted in General Code/Help
      K
      kriku
    • RE: Maximum number of data lines?

      Data processing cost 4 years 1 stock hourly bars: 15.94 sec
      Without any indicators & instant return from next(): 6.03 sec
      With all indicators present & instant return from next(): 14.02 sec

      Therefore, the usage of 8 indicators costs about 8 secs in my case. Out of these, the most costly one is the StandardDeviation: 2.7 secs.

      NB! These figures are just indicative since I measured the performance only once.

      The code in the article referenced above uses just 2 moving averages as indicators. My script finishes in 6.36 secs when I run it with the following indicators only:

      mvav1 = bt.indicators.MovingAverageSimple(data, period=300)
      mvav2 = bt.indicators.MovingAverageSimple(data, period=600)
      

      The two moving averages together cost about half a sec or 16x less than the 8 indicators in the real code (and more than 5x less than the StandardDeviation).

      This was done with Python 3.7, NOT pypy. As the article says that using pypy could more than double performance, I am beginning to think that the performance of my script is presently close to normal.

      Does all this make more sense to you?

      posted in General Code/Help
      K
      kriku
    • RE: Maximum number of data lines?

      Thanks for the good tip.

      I finally got time to have a serious look at the issue. As I suspected, it had nothing to do with the Backtrader. pandas_market_calendars.get_calendar("NASDAQ").valid_days(start_date=start_dt, end_date=end_dt) costs as much as 0.14+ seconds for some odd reason. The script is a little more than 10 times faster now.

      That means, it takes about 44 seconds for one stock from 1.1.2010 to 1.1.2019. I reckon that it is still slow by your standards, but I do not think it is BT's fault in any way. I will try to find out more in the nearest future.

      posted in General Code/Help
      K
      kriku
    • RE: Maximum number of data lines?

      In that case, I must be doing something very wrong indeed.

      I suspect that it is my usage of the Indicators & Observers. Right now I am creating them separately for each data feed similarly to following:

      class MyStrategy(bt.Strategy):
      
        my_indicators = {}
      
        def __init__(self):
      
          for data in self.datas:
            symbol = data.p.dataname.split("\\")[1].split(".")[0]
            self.my_indicators[symbol] = MyIndicator(data)
      

      I create an Observer for each feed principally in the same way (in the main, ofc).

      It did not look quite right when I wrote it but with a small number of feeds, it did not matter much.

      Is this to blame and if yes, what is the right way to use the Indicators with multiple data feeds?

      posted in General Code/Help
      K
      kriku
    • Maximum number of data lines?

      Is there a maximum number of data lines (different equities to trade simultaneously) that Backtrader can reasonably handle?

      Right now I am using hourly data for 215 stocks for 4 years. The completion of the backtest takes several hours. This is all right but I wonder how much data could be added. Would it still work with 500 stocks? 1000 stocks? 2000?

      Is there a formula to predict the performance basing on the amount of data?

      I reckon that the answer also depends on the number of indicators & observers and their complexity, but still, if the backtest takes, say, 5 hours with 215 stocks, should it be near 50 for 2150 stocks or not?

      I have 16 GB of RAM so that should not be an obstacle.

      Thanks in advance.

      posted in General Code/Help
      K
      kriku
    • RE: Error while retrieving IBStore historic data with an IB paper account

      That's right. This works:

      data = ibstore.getdata(dataname="AAPL-STK-SMART-USD", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1), timeframe=bt.TimeFrame.Minutes, compression=5)
      

      I am sorry for having been a nuisance. BT is an impressive project and the forums show that you provide prompt and straight-to-the-point support even to stupid questions. Keep up the good work!

      posted in General Code/Help
      K
      kriku
    • RE: Error while retrieving IBStore historic data with an IB paper account

      Thanks, I had completely forgotten about this setting.

      The field was empty. However, I filled it with 6666666 and now BOTH of the code snippets in my last post raise the same Exception while a similar request with clientId = 7777777 does not. That's a bit odd but goes beyond the scope of the Backtrader discussion.

      However, I still cannot retrieve the historical data and I cannot understand what I am doing wrong. Here is the whole code (based on the First Strategy tutorial):

      from _datetime import datetime
      import backtrader as bt
      
      # Create a Stratey
      class TestStrategy(bt.Strategy):
      
          def log(self, txt, dt=None):
              ''' Logging function for this strategy'''
              dt = dt or self.datas[0].datetime.date(0)
              print('%s, %s' % (dt.isoformat(), txt))
      
          def __init__(self):
              # Keep a reference to the "close" line in the data[0] dataseries
              self.dataclose = self.datas[0].close
      
          def next(self):
              # Simply log the closing price of the series from the reference
              self.log('Close, %.2f' % self.dataclose[0])
      
      
      if __name__ == '__main__':
          # Create a cerebro entity
          cerebro = bt.Cerebro()
      
          ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=7777777)
          #ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666)
          data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                  timeframe=bt.TimeFrame.Minutes, compression=5)
          cerebro.adddata(data)
      
          cerebro.addstrategy(TestStrategy)
      
          cerebro.broker.setcash(100000.0)
      
          # Print out the starting conditions
          print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
      
          # Run over everything
          cerebro.run()
      
          # Print out the final result
          print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())
      

      The output is:

      Starting Portfolio Value: 100000.00
      Server Version: 76
      TWS Time at connection:20190817 12:16:29 
      Final Portfolio Value: 100000.00
      

      The breakpoint in the next() is never reached, no Exceptions are raised.

      To be sure about the permissions, I just downloaded the 5 minute AAPL bars for the last 2 years with my own client.

      Many thanks for your time.

      posted in General Code/Help
      K
      kriku
    • RE: Error while retrieving IBStore historic data with an IB paper account

      Yes, I know I am confused. Nothing to be sorry about. :) Thanks for the clarification about the clientId.

      However, I am getting even more confused. The outcomes of using my account ID for the clientId and an arbitrary number such as 666 differ. How could THAT be???

      This results in the exception described in my first post:

      ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=105****)  # my account number, digits replaced with asterisks
      data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
           timeframe=bt.TimeFrame.Minutes, compression=5)
      cerebro.adddata(data)
      

      This does not raise an exception but does not retrieve any data either:

      ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=6666666)
      data = ibstore.getdata(dataname="AAPL", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
          timeframe=bt.TimeFrame.Minutes, compression=5)
      cerebro.adddata(data)
      
      posted in General Code/Help
      K
      kriku
    • RE: Error while retrieving IBStore historic data with an IB paper account

      In the beginning, I used my real account ID which I would not like to disclose here. It is a 7-digit number: 105xxxx. That resulted in the Exception from the BackBroker. I am sorry that I failed to indicate it in my first post.

      Later, I tried a fictional client id such as 666 to see what happens.

      It is not an issue of permissions since I do get the historic data with a test client that I wrote from scratch with the same clientId, including AAPL. Yes, I am using the GW.

      The exception is raised regardless I use "aapl" or "AAPL". I suspect that it happens before the historical data download.

      posted in General Code/Help
      K
      kriku
    • RE: Error while retrieving IBStore historic data with an IB paper account

      I was too optimistic. It does not raise an exception but it neither retrieves any data. It would be surprising if it did with a fictional client id.

      Please help.

      posted in General Code/Help
      K
      kriku
    • RE: Error while retrieving IBStore historic data with an IB paper account

      It seems to be that it was a mistake to query with a real client ID. That works:

      ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=666)    
      data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                                           timeframe=bt.TimeFrame.Minutes, compression=5)
      cerebro.adddata(data)
      
      # Set our desired cash start
      cerebro.broker.setcash(100000.0)
      
      # Print out the starting conditions
      print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())
      
      # Run over everything
      cerebro.run()
      
      posted in General Code/Help
      K
      kriku
    • Error while retrieving IBStore historic data with an IB paper account

      Hello

      I connect to an existing paper account as following:

      ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=cid)
      data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                                           timeframe=bt.TimeFrame.Minutes, compression=5)
      cerebro.adddata(data)
      

      I get the following error:

      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
          self.broker.push_orderstate(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstate'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
          self.broker.push_orderstatus(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
          self.broker.push_orderstate(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstate'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
          self.broker.push_orderstatus(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
          self.broker.push_orderstate(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstate'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
          self.broker.push_orderstatus(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'openOrder' for 'openOrder'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1291, in openOrder
          self.broker.push_orderstate(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstate'
      14-Aug-19 12:24:24 ERROR     Exception in message dispatch.  Handler 'orderStatus' for 'orderStatus'
      Traceback (most recent call last):
        File "C:\Python37-64\lib\site-packages\ib\opt\dispatcher.py", line 44, in __call__
          results.append(listener(message))
        File "C:\Users\ksander\correlation\backtrader\stores\ibstore.py", line 1301, in orderStatus
          self.broker.push_orderstatus(msg)
      AttributeError: 'NoneType' object has no attribute 'push_orderstatus'
      

      It seems to be that the server starts pushing order status etc to the client and the BackBroker does not know what to do with them. Quite logical. When I add an IBBroker as following:

      ibstore = bt.stores.IBStore(host='127.0.0.1', port=4002, clientId=cid)
      cerebro.broker = ibstore.getbroker()
      data = ibstore.getdata(dataname="aapl", historical=True, fromdate=datetime(2019, 1, 1), todate=datetime(2019, 8, 1),
                                           timeframe=bt.TimeFrame.Minutes, compression=5)
      cerebro.adddata(data)
      

      that error is gone but I will soon get another one:

      Traceback (most recent call last):
        File "C:/Users/ksander/correlation/bt_katse.py", line 66, in <module>
          cerebro.broker.setcash(100000.0)
      AttributeError: 'IBBroker' object has no attribute 'setcash'
      

      Also logical - the live broker cannot let you set your cash indeed :)

      The question is, what is the proper way to download the historical data from the IB if I cannot use an existing paper account with standing orders etc because of that issue?

      Does the BackBroker need some exception handling?

      Thanks in advance.

      posted in General Code/Help
      K
      kriku
    • 1 / 1