Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Optimzation question

    General Code/Help
    2
    9
    3222
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      asuralm last edited by

      Hi:

      I was following the post [https://community.backtrader.com/topic/125/strategy-auto-generation](link url) and it is very interesting to me.

      Here is the code that I try to test the idea.
      from future import (absolute_import, division, print_function,
      unicode_literals)
      import itertools
      import datetime
      import backtrader as bt

      class Entry1(bt.Indicator):
          lines = ('HighN',)
          params = (('para1', 10),)
          def __init__(self):
              self.lines.HighN = bt.ind.Highest(self.p.para1)
      
      class Entry2(bt.Indicator):
          lines = ('longMA', )
          params = (('para1', 30),)
          def __init__(self):
              self.lines.longsig = self.data.close > bt.ind.SMA(period=self.p.para1)
      
      class Exit1(bt.Indicator):
          lines = ('LowN',)
          params = (('para1', 10),)
          def __init__(self):
              self.lines.LowN = bt.ind.Lowest(self.p.para1)
      
      class Exit2(bt.Indicator):
          lines = ('shortMA', )
          params = (('para1', 30),)
          def __init__(self):
              self.lines.longsig = self.data.close < bt.ind.SMA(period=self.p.para1)
      
      class MasterStrategy(bt.Strategy):
          params = (('entry', None), ('exit', None),('para1', 10),)
      
          def __init__(self):
              print("in init")
              self.entry = self.p.entry
              self.exit = self.p.exit
              self.para1 = self.p.para1
              print("entry=", self.p.entry)
              print("exit=", self.p.exit)
              print("para1=", self.p.para1)
      
          def stop(self):
              print("strategy stop")
          def next(self):
              pass  # do the buy/sell logic here
      
      if __name__ == '__main__':
          signals = itertools.product((Entry1, Entry2), (Exit1, Exit2))
          for entry, exit in signals:
              print("Running entry=", entry, " exit=", exit)
              cerebro = bt.Cerebro()
      
              fromdate = datetime.datetime(2014,1,1)
              todate = datetime.datetime(2015,1,1)
              # Load data
              data = bt.feeds.GenericCSVData(dataname='a#_m15_processed.csv',
                      dtformat='%Y-%m-%d', fromdate=fromdate, todate=todate,
                     datetime=0, time=1, open=2, high=3, low=4, close=5, volume=6, openinterest=8, timeframe=bt.TimeFrame.Minutes)
              cerebro.adddata(data)
              cerebro.optstrategy(MasterStrategy, entry=entry, exit=exit, para1=[10,20])
              cerebro.run()
      

      However, when I run the code, it gives me
      Connected to pydev debugger (build 172.3968.37)
      Running entry= <class 'main.Entry1'> exit= <class 'main.Exit1'>
      in init
      entry= <class 'mp_main.Entry1'>
      exit= <class 'mp_main.Exit1'>
      para1= 10
      in init
      entry= <class 'mp_main.Entry1'>
      exit= <class 'mp_main.Exit1'>
      para1= 20
      strategy stop
      Exception in thread Thread-16:
      Traceback (most recent call last):
      File "D:\ProgramData\Anaconda3_4.4.0\lib\threading.py", line 916, in _bootstrap_inner
      self.run()
      File "D:\ProgramData\Anaconda3_4.4.0\lib\threading.py", line 864, in run
      self._target(*self._args, **self._kwargs)
      File "D:\ProgramData\Anaconda3_4.4.0\lib\multiprocessing\pool.py", line 429, in _handle_results
      task = get()
      File "D:\ProgramData\Anaconda3_4.4.0\lib\multiprocessing\connection.py", line 251, in recv
      return _ForkingPickler.loads(buf.getbuffer())
      AttributeError: Can't get attribute 'Entry1' on <module 'pydevd' from 'D:\Program Files\JetBrains\PyCharm Community Edition 2017.2.2\helpers\pydev\pydevd.py'>

      strategy stop


      And the code stuck in there like an infinite loop.
      I tried addstrategy instead of optstrategy and it work fine.
      What's the error in this code please?

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by backtrader

        The code above cannot be properly read. But if you are looking at the post Community - https://community.backtrader.com/topic/125/strategy-auto-generation/16, look at the specif post for a working script here: "@ab_trader said in Strategy auto-generation"

        How indicators are passed properly to a strategy is there, rather than looking at hand-crafted non-tested scripts.

        # entries
        class Entry_XXX(bt.Indicator):
            lines = ('longsig', 'shortsig')
            params = (('XXX_param1', 5), ('XXX_param2', 10),)
        
            def __init__(self):
                self.lines.longsig = # function of XXX_param1 & XXX_param2
                self.lines.shortsig = # other function of XXX_param1 & XXX_param2
        
        # exits
        class Exit_YYY(bt.Indicator):
            lines = ('sellsig', 'coversig')
            params = (('YYY_param1', 5), ('YYY_param2', 10), ('YYY_param3', 2),)
        
            def __init__(self):
                self.lines.sellsig = # function of YYY_param1, YYY_param2 & YYY_param3
                self.lines.coversig = # other function of YYY_param1, YYY_param2 & YYY_param3
        
        # strategy
        class MasterStrategy(bt.Strategy):
        
            params = (('entry', None), ('exit', None), ('support', None),
                      ('plot_entry', True), ('plot_exit', True), ('plot_support', True),)
        
            def __init__(self):
                self.longsig = None
                self.shortsig = None
                self.sellsig = None
                self.coversig = None
        
                if self.p.entry:
                    self.entry = self.p.entry(plot=self.p.plot_entry)
                    self.longsig = self.entry.lines.longsig
                    self.shortsig = self.entry.lines.shortsig
        
                if self.p.exit:
                    self.exit = self.p.exit(plot=self.p.plot_exit)
                    self.sellsig = self.exit.lines.sellsig
                    self.coversig = self.exit.lines.coversig
        
            def next(self):
                # process buy, short, sell and cover signals
        
        #main code
        if __name__ == '__main__':
            cerebro = bt.Cerebro()
            strats = cerebro.optstrategy(MasterStrategy,
                entry = (Entry_1, Entry_2, Entry_XXX),
                exit = (Exit_1, Exit_2, Exit_YYY)
                )
            data = bt.feeds.YahooFinanceCSVData(Parameters set as usual)
            cerebro.adddata(data)
            cerebro.run()
        
        A 1 Reply Last reply Reply Quote 0
        • A
          asuralm @backtrader last edited by

          @backtrader

          Hi backtrader, I tried the script you gave me above, I just encounter the exactly same error and the program runs into a dead loop with no response.

          the output is like:

          "D:\Program Files\Python36\python.exe" "D:\Program Files\JetBrains\PyCharm Community Edition 2017.1.2\helpers\pydev\pydevd.py" --multiproc --qt-support --client 127.0.0.1 --port 50825 --file "C:/Users/A730-Home/Documents/python codes/strategyoptimization/StrategyMaster.py"
          pydev debugger: process 189624 is connecting
          
          Connected to pydev debugger (build 171.4249.47)
          in entry init
           in exit init 
          in the initialization
          Exception in thread Thread-16:
          Traceback (most recent call last):
            File "D:\Program Files\Python36\lib\threading.py", line 916, in _bootstrap_inner
              self.run()
            File "D:\Program Files\Python36\lib\threading.py", line 864, in run
              self._target(*self._args, **self._kwargs)
            File "D:\Program Files\Python36\lib\multiprocessing\pool.py", line 429, in _handle_results
              task = get()
            File "D:\Program Files\Python36\lib\multiprocessing\connection.py", line 251, in recv
              return _ForkingPickler.loads(buf.getbuffer())
          AttributeError: Can't get attribute 'Entry_XXX' on <module 'pydevd' from 'D:\\Program Files\\JetBrains\\PyCharm Community Edition 2017.1.2\\helpers\\pydev\\pydevd.py'>
          

          My code :

          from __future__ import(absolute_import, division, print_function, unicode_literals)
          
          import datetime  # For datetime objects
          import backtrader as bt
          # entries
          class Entry_XXX(bt.Indicator):
              lines = ('longsig', 'shortsig')
              params = (('XXX_param1', 5), ('XXX_param2', 10),)
          
              def __init__(self):
                  self.lines.longsig = self.datas[0].close > self.datas[0].low
                  self.lines.shortsig = self.datas[0].close < self.datas[0].low
                  print("in entry init")
          
          # exits
          class Exit_YYY(bt.Indicator):
              lines = ('sellsig', 'coversig')
              params = (('YYY_param1', 5), ('YYY_param2', 10), ('YYY_param3', 2),)
          
              def __init__(self):
                  self.lines.sellsig = self.datas[0].close > self.datas[0].low
                  self.lines.coversig = self.datas[0].close > self.datas[0].low
                  print(" in exit init ")
          
          # strategy
          class MasterStrategy(bt.Strategy):
          
              params = (('entry', None), ('exit', None), ('support', None),
                        ('plot_entry', True), ('plot_exit', True), ('plot_support', True),)
          
              def __init__(self):
                  self.longsig = None
                  self.shortsig = None
                  self.sellsig = None
                  self.coversig = None
          
                  if self.p.entry:
                      self.entry = self.p.entry(plot=self.p.plot_entry)
                      self.longsig = self.entry.lines.longsig
                      self.shortsig = self.entry.lines.shortsig
          
                  if self.p.exit:
                      self.exit = self.p.exit(plot=self.p.plot_exit)
                      self.sellsig = self.exit.lines.sellsig
                      self.coversig = self.exit.lines.coversig
          
                  print("in the initialization")
          
              def next(self):
                  # process buy, short, sell and cover signals
                  pass
          
          #main code
          if __name__ == '__main__':
              cerebro = bt.Cerebro()
              strats = cerebro.optstrategy(MasterStrategy,
                  entry = (Entry_XXX,),
                  exit = (Exit_YYY,)
                  )
              # Load data
              data = bt.feeds.GenericCSVData(
                  dataname='../v#_m15_processed.csv',
                  dtformat='%Y-%m-%d', fromdate=datetime.datetime(2014,1,1), todate=datetime.datetime(2015,1,1),
                  datetime=0, time=1, open=2, high=3, low=4, close=5, volume=6, openinterest=8, timeframe=bt.TimeFrame.Minutes)
              cerebro.adddata(data)    
              cerebro.run()
          
          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            @asuralm said in Optimzation question:

            pydevd

            AttributeError: Can't get attribute 'Entry_XXX' on <module 'pydevd'
            

            vs

            if __name__ == '__main__':
            

            It would seem your environment is in play.

            Execute it from the console.

            A 1 Reply Last reply Reply Quote 0
            • A
              asuralm @backtrader last edited by

              @backtrader
              Thanks it works fine in the shell.

              It's a shame not working well in pycharm.

              1 Reply Last reply Reply Quote 0
              • B
                backtrader administrators last edited by

                Environments like PyCharm, Spyder, Jupyter and possibly others don't work with the multiprocessing module, because they hijack the start of the Python Kernel, which prevents the proper initialization (under Windows mostly)

                This is described (for example) in:

                • Community - FAQ
                • Stack Overflow - Multithreading frozen in Python Spyder but not command prompt Windows 10
                A 1 Reply Last reply Reply Quote 0
                • A
                  asuralm @backtrader last edited by

                  @backtrader
                  thanks.

                  I just find a way to bypass the error. Simple enforce the strategy optimization to single thread as:

                  cerebro.optstrategy(MasterStrategy, entry=entry, exit=exit, para1=[10,20])
                  cerebro.run(maxcpus=1)
                  

                  Then IDE works with no error. At least I can debug in IDE and when I can switch to multi-threading and run in IDLE after testing.

                  1 Reply Last reply Reply Quote 0
                  • A
                    asuralm last edited by

                    @backtrader
                    There is a question about the optimization improvement. I've read the article optimization-improvements, but it seems that the improvement works in the scenario of single strategy optimization.

                    In this strategy auto-generation case, the problem is that trading data are the same and strategy is switching. Is this possible that the data file is only read-once and switching between strategies only?

                    My current structure is like:

                        signals = itertools.product(EntryList, ExitList)
                        for sig, exit in signals:
                            cerebro = bt.Cerebro()
                            fromdate = datetime.datetime(2014, 1, 1)
                            todate = datetime.datetime(2015, 1, 1)
                            data = bt.feeds.GenericCSVData(...)
                            cerebro.adddata(data)
                    
                            cerebro.optstrategy(MasterStrategy, signal=sig, exsig1=exit, [paras]...)
                            cerebro.run(maxcpus=1)
                    

                    In such structure, the data is loaded every time in the loop and file reading is very time-consuming.
                    It seems not working if I simply move the code before adddata out of the for loop.
                    Is it possible that the file is read only once?

                    Have you got any good suggestions to improve the performance please?

                    Thanks

                    1 Reply Last reply Reply Quote 0
                    • B
                      backtrader administrators last edited by

                      Cerebro doesn't know that you are looping and cannot know it. The runs inside the optimization re-use the file which has already been loaded, but the runs doesn't know that you had another set of runs before with the same file.

                      You could try to keep the data feed outside of the loop and execute a data.home() after the cerebro.run(...) is done to reset the pointer.

                      Completely untested and may absolutely not work.

                      1 Reply Last reply Reply Quote 0
                      • 1 / 1
                      • First post
                        Last post
                      Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors