Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Optimizing with IBStore causes redundant connections/downloads

    General Code/Help
    3
    6
    1952
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      d416 last edited by d416

      I'm trying to optimize a Simple Moving Average cross-over strategy using Interactive Brokers data via IBStore.

      While optimizing, the processes initiate numerous connects/disconnects to the IB Trader Workstation as the parameters are iterated - that is, one connection for each permutation of parameters. Both the TWS data window shows this, and a similar number of "TWS Time at connection:20170514 16:06:03 EST" lines are printed on the console. (If the optimization is iterating through 100 permutations, there are 100 connections)

      I'm guessing that with each iteration, no matter what the data source, the data is loaded into the strategy? I figured the default values for preload/runonce/optdatas would handle this.

      If this is not the case, is the proper way to do this just pre-load the required data set into Pandas or influxdb and use it from there?

      from __future__ import (absolute_import, division, print_function,
                          unicode_literals)
      
      import datetime  # For datetime objects
      import os.path  # To manage paths
      import sys  # To find out the script name (in argv[0])
      
      # Import the backtrader platform
      import backtrader as bt
      
      # Create a Data Feed
      ibstore = bt.stores.IBStore(host='127.0.0.1',port=7496,clientId=5)
      mysymbol = 'EUR.USD-CASH-IDEALPRO'
      fromdate = datetime.datetime(2015,1,1,00,00)
      todate = fromdate = datetime.datetime(2017,3,30,00,00)
      data = ibstore.getdata(dataname=mysymbol, timeframe=bt.TimeFrame.Days, compression=1, fromdate=fromdate, todate=todate, historical=True)
      
      
      # Create a Strategy
      class SmaCross(bt.SignalStrategy):
      params = (
          ('sma1', 10),
          ('sma2', 30),
      )
      def __init__(self):
          SMA1 = bt.ind.SMA(period=int(self.params.sma1))
          SMA2 = bt.ind.SMA(period=int(self.params.sma2))
          crossover = bt.ind.CrossOver(SMA1, SMA2)
          self.signal_add(bt.SIGNAL_LONG, crossover)
      
      
      
      if __name__ == '__main__':
      # Create a cerebro entity
      cerebro = bt.Cerebro(maxcpus=1)
      
      # Add a strategy
      strats = cerebro.optstrategy(
          SmaCross,
          sma1=range(50, 60),
          sma2=range(50,60))
      
      # Add the Data Feed to Cerebro
      cerebro.adddata(data)
      
      # Set our desired cash start
      cerebro.broker.setcash(10000.0)
      
      # Add a FixedSize sizer according to the stake
      cerebro.addsizer(bt.sizers.FixedSize, stake=1000)
      
      # Set the commission
      cerebro.broker.setcommission(commission=0.0)
      
      # Run over everything
      cerebro.run()
      
      D 1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        Optimization was in the platform long before any kind of live streams was added and will simply ask the streams to load its data if the data is not preloaded.

        In this case the data is not seen as preloaded because of the live nature of the feed and that's why it is being recollected.

        It was never envisioned (also taking into account the historical download limitations inherent to IB) that such feeds would be used in an optimization attempt.

        D 1 Reply Last reply Reply Quote 1
        • D
          d416 @backtrader last edited by

          @backtrader Than you for the confirm (and this gift of software!)

          It seems the answer then is to

          1. Get a snapshot of IB historical data and load into a more optimal store, such as a Pandas data frame feed
          2. Use the Pandas data frame as a data feed in the optimization

          On point 1, is there an easy way you can think of to copy the ibstore data into a Pandas data frame? Or perhaps this would best be handled using the external IBpy methods to prep the data.

          -D

          1 Reply Last reply Reply Quote 0
          • D
            d416 @d416 last edited by

            Resolved: see this post for a solution...

            https://community.backtrader.com/topic/447/code-snippet-copying-ib-data-to-pandas-for-efficiency

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators last edited by

              The message above got under the radar. A potential solution (untested) would be:

              class MyData(bt.feeds.IBData):
                  def islive(self):
                      '''Usually Returns ``True`` to notify ``Cerebro`` that preloading and runonce
                      should be deactivated
              
                      --- Returns False in subclass to allow optimization
                      '''
                      return False
              

              and then

              bt.stores.IBStore.DataCls = MyData
              
              R 1 Reply Last reply Reply Quote 0
              • R
                RY93 @backtrader last edited by

                @backtrader
                I have a similar issue.
                I am trying to run optimization on IB data(live or historical) source.
                When using optStrategy I am getting this error(attached below) ,but with other data sources(csv file using GenericCSVData) it is running fine.
                Even In addstrategy method this IB data source works , thus it seems to me the problem is using IB data source with optStrategy only.

                Traceback (most recent call last):
                  File "D:/Users/R/BackTestLiveIBv2.py", line 219, in <module>
                    runstrategy()
                  File "D:/Users/R/BackTestLiveIBv2.py", line 216, in runstrategy
                    cerebro.run()
                  File "C:\Users\R\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1143, in run
                    for r in pool.imap(self, iterstrats):
                  File "C:\Users\R\Anaconda3\lib\multiprocessing\pool.py", line 735, in next
                    raise value
                  File "C:\Users\R\Anaconda3\lib\multiprocessing\pool.py", line 424, in _handle_tasks
                    put(task)
                  File "C:\Users\R\Anaconda3\lib\multiprocessing\connection.py", line 206, in send
                    self._send_bytes(_ForkingPickler.dumps(obj))
                  File "C:\Users\R\Anaconda3\lib\multiprocessing\reduction.py", line 51, in dumps
                    cls(buf, protocol).dump(obj)
                TypeError: can't pickle _thread.lock objects
                

                Here is the code :

                import argparse
                import datetime
                import backtrader as bt
                from backtrader.utils import flushfile  
                import os
                
                class TestStrategy(bt.Strategy):
                
                    params = dict(
                        smaperiod = 5,
                        trade=False,
                        stake=10,
                        exectype=bt.Order.Market,
                        stopafter=0,
                        valid=None,
                        cancel=0,
                        donotsell=False,
                        optim=False,
                        optimParams= (0, 0),
                    )
                
                    def __init__(self):
                        # To control operation entries
                        self.orderid = list()
                        self.order = None
                
                        self.counttostop = 0
                        self.datastatus = 0
                
                        # Create SMA on 2nd data
                        if self.p.optim:  # Use a tuple during optimization
                            self.p.smaperiod,self.p.stake = self.p.optimParams
                        self.sma = bt.indicators.MovAv.SMA(self.data, period=self.p.smaperiod)
                
                
                    def notify_data(self, data, status, *args, **kwargs):
                        print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)
                        if status == data.LIVE:
                            self.counttostop = self.p.stopafter
                            self.datastatus = 1
                
                    def notify_order(self, order):
                        if order.status in [order.Completed, order.Cancelled, order.Rejected]:
                            self.order = None
                
                        print('-' * 50, 'ORDER BEGIN', datetime.datetime.now())
                        # print(order)
                        print('-' * 50, 'ORDER END')
                
                    def notify_trade(self, trade):
                        print('-' * 50, 'TRADE BEGIN', datetime.datetime.now())
                        print(trade)
                        print('-' * 50, 'TRADE END')
                
                    def prenext(self):
                        self.next(frompre=True)
                
                    def next(self, frompre=False):
                
                        txt = list()
                        txt.append('%04d' % len(self))
                        dtfmt = '%Y-%m-%d %H:%M:%S.%f'
                        txt.append('%s' % self.data.datetime.datetime(0).strftime(dtfmt))
                        txt.append('{}'.format(self.data.open[0]))
                        txt.append('{}'.format(self.data.high[0]))
                        txt.append('{}'.format(self.data.low[0]))
                        txt.append('{}'.format(self.data.close[0]))
                        txt.append('{}'.format(self.data.volume[0]))
                        txt.append('{}'.format(self.data.openinterest[0]))
                        txt.append('{}'.format(self.sma[0]))
                        print(', '.join(txt))
                
                
                        if self.counttostop:  # stop after x live lines
                            self.counttostop -= 1
                            if not self.counttostop:
                                self.env.runstop()
                                return
                
                        if not self.p.trade:
                            return
                
                        if self.datastatus and not self.position and len(self.orderid) < 1:
                            self.order = self.buy(size=self.p.stake,
                                                  exectype=self.p.exectype,
                                                  price=round(self.data0.close[0] * 0.90, 2),
                                                  valid=self.p.valid)
                
                            self.orderid.append(self.order)
                        elif self.position.size > 0 and not self.p.donotsell:
                            if self.order is None:
                                self.order = self.sell(size=self.p.stake // 2,
                                                       exectype=bt.Order.Market,
                                                       price=self.data0.close[0])
                
                        elif self.order is not None and self.p.cancel:
                            if self.datastatus > self.p.cancel:
                                self.cancel(self.order)
                
                        if self.datastatus:
                            self.datastatus += 1
                
                    def start(self):
                
                        header = ['Datetime', 'Open', 'High', 'Low', 'Close', 'Volume',  'OpenInterest', 'SMA']
                        print(', '.join(header))
                
                        self.done = False
                
                
                def runstrategy():
                
                    cerebro = bt.Cerebro(stdstats=False)
                
                    storekwargs = dict(host='127.0.0.1',
                                       port=7497,
                                       clientId=0, 
                                       notifyall=False, 
                                       _debug=False,
                                       reconnect = 3,  
                                       timeout = 3, 
                                       timeoffset =  False,  
                                       timerefresh = 60.0,  
                                       )
                
                    store = bt.stores.IBStore(**storekwargs)
                
                    datakwargs = dict(
                        timeframe=bt.TimeFrame.Seconds,
                        compression=1,
                        historical= True,
                        rtbar=False,  # real time bars
                        qcheck=0.5,
                        backfill_start=True,
                        backfill=True,
                        latethrough=True,
                        tz='GMT',
                    )
                
                    data = store.getdata(dataname='USDINR-IND-NSE-INR', **datakwargs)
                    cerebro.resampledata(dataname= data)
                
                    # Add the strategy
                    # cerebro.addstrategy(TestStrategy,
                    #                     smaperiod = 5,
                    #                     trade = True)
                
                    cerebro.optstrategy(TestStrategy,
                                        optim=True,
                                        optimParams=[[5,10],[10,10]],
                                        trade=True)
                    cerebro.run()
                
                if __name__ == '__main__':
                    runstrategy()
                
                
                1 Reply Last reply Reply Quote 0
                • 1 / 1
                • First post
                  Last post
                Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors