Optimzation question



  • Hi:

    I was following the post [https://community.backtrader.com/topic/125/strategy-auto-generation](link url) and it is very interesting to me.

    Here is the code that I try to test the idea.
    from future import (absolute_import, division, print_function,
    unicode_literals)
    import itertools
    import datetime
    import backtrader as bt

    class Entry1(bt.Indicator):
        lines = ('HighN',)
        params = (('para1', 10),)
        def __init__(self):
            self.lines.HighN = bt.ind.Highest(self.p.para1)
    
    class Entry2(bt.Indicator):
        lines = ('longMA', )
        params = (('para1', 30),)
        def __init__(self):
            self.lines.longsig = self.data.close > bt.ind.SMA(period=self.p.para1)
    
    class Exit1(bt.Indicator):
        lines = ('LowN',)
        params = (('para1', 10),)
        def __init__(self):
            self.lines.LowN = bt.ind.Lowest(self.p.para1)
    
    class Exit2(bt.Indicator):
        lines = ('shortMA', )
        params = (('para1', 30),)
        def __init__(self):
            self.lines.longsig = self.data.close < bt.ind.SMA(period=self.p.para1)
    
    class MasterStrategy(bt.Strategy):
        params = (('entry', None), ('exit', None),('para1', 10),)
    
        def __init__(self):
            print("in init")
            self.entry = self.p.entry
            self.exit = self.p.exit
            self.para1 = self.p.para1
            print("entry=", self.p.entry)
            print("exit=", self.p.exit)
            print("para1=", self.p.para1)
    
        def stop(self):
            print("strategy stop")
        def next(self):
            pass  # do the buy/sell logic here
    
    if __name__ == '__main__':
        signals = itertools.product((Entry1, Entry2), (Exit1, Exit2))
        for entry, exit in signals:
            print("Running entry=", entry, " exit=", exit)
            cerebro = bt.Cerebro()
    
            fromdate = datetime.datetime(2014,1,1)
            todate = datetime.datetime(2015,1,1)
            # Load data
            data = bt.feeds.GenericCSVData(dataname='a#_m15_processed.csv',
                    dtformat='%Y-%m-%d', fromdate=fromdate, todate=todate,
                   datetime=0, time=1, open=2, high=3, low=4, close=5, volume=6, openinterest=8, timeframe=bt.TimeFrame.Minutes)
            cerebro.adddata(data)
            cerebro.optstrategy(MasterStrategy, entry=entry, exit=exit, para1=[10,20])
            cerebro.run()
    

    However, when I run the code, it gives me
    Connected to pydev debugger (build 172.3968.37)
    Running entry= <class 'main.Entry1'> exit= <class 'main.Exit1'>
    in init
    entry= <class 'mp_main.Entry1'>
    exit= <class 'mp_main.Exit1'>
    para1= 10
    in init
    entry= <class 'mp_main.Entry1'>
    exit= <class 'mp_main.Exit1'>
    para1= 20
    strategy stop
    Exception in thread Thread-16:
    Traceback (most recent call last):
    File "D:\ProgramData\Anaconda3_4.4.0\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
    File "D:\ProgramData\Anaconda3_4.4.0\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
    File "D:\ProgramData\Anaconda3_4.4.0\lib\multiprocessing\pool.py", line 429, in _handle_results
    task = get()
    File "D:\ProgramData\Anaconda3_4.4.0\lib\multiprocessing\connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
    AttributeError: Can't get attribute 'Entry1' on <module 'pydevd' from 'D:\Program Files\JetBrains\PyCharm Community Edition 2017.2.2\helpers\pydev\pydevd.py'>

    strategy stop


    And the code stuck in there like an infinite loop.
    I tried addstrategy instead of optstrategy and it work fine.
    What's the error in this code please?


  • administrators

    The code above cannot be properly read. But if you are looking at the post Community - https://community.backtrader.com/topic/125/strategy-auto-generation/16, look at the specif post for a working script here: "@ab_trader said in Strategy auto-generation"

    How indicators are passed properly to a strategy is there, rather than looking at hand-crafted non-tested scripts.

    # entries
    class Entry_XXX(bt.Indicator):
        lines = ('longsig', 'shortsig')
        params = (('XXX_param1', 5), ('XXX_param2', 10),)
    
        def __init__(self):
            self.lines.longsig = # function of XXX_param1 & XXX_param2
            self.lines.shortsig = # other function of XXX_param1 & XXX_param2
    
    # exits
    class Exit_YYY(bt.Indicator):
        lines = ('sellsig', 'coversig')
        params = (('YYY_param1', 5), ('YYY_param2', 10), ('YYY_param3', 2),)
    
        def __init__(self):
            self.lines.sellsig = # function of YYY_param1, YYY_param2 & YYY_param3
            self.lines.coversig = # other function of YYY_param1, YYY_param2 & YYY_param3
    
    # strategy
    class MasterStrategy(bt.Strategy):
    
        params = (('entry', None), ('exit', None), ('support', None),
                  ('plot_entry', True), ('plot_exit', True), ('plot_support', True),)
    
        def __init__(self):
            self.longsig = None
            self.shortsig = None
            self.sellsig = None
            self.coversig = None
    
            if self.p.entry:
                self.entry = self.p.entry(plot=self.p.plot_entry)
                self.longsig = self.entry.lines.longsig
                self.shortsig = self.entry.lines.shortsig
    
            if self.p.exit:
                self.exit = self.p.exit(plot=self.p.plot_exit)
                self.sellsig = self.exit.lines.sellsig
                self.coversig = self.exit.lines.coversig
    
        def next(self):
            # process buy, short, sell and cover signals
    
    #main code
    if __name__ == '__main__':
        cerebro = bt.Cerebro()
        strats = cerebro.optstrategy(MasterStrategy,
            entry = (Entry_1, Entry_2, Entry_XXX),
            exit = (Exit_1, Exit_2, Exit_YYY)
            )
        data = bt.feeds.YahooFinanceCSVData(Parameters set as usual)
        cerebro.adddata(data)
        cerebro.run()
    


  • @backtrader

    Hi backtrader, I tried the script you gave me above, I just encounter the exactly same error and the program runs into a dead loop with no response.

    the output is like:

    "D:\Program Files\Python36\python.exe" "D:\Program Files\JetBrains\PyCharm Community Edition 2017.1.2\helpers\pydev\pydevd.py" --multiproc --qt-support --client 127.0.0.1 --port 50825 --file "C:/Users/A730-Home/Documents/python codes/strategyoptimization/StrategyMaster.py"
    pydev debugger: process 189624 is connecting
    
    Connected to pydev debugger (build 171.4249.47)
    in entry init
     in exit init 
    in the initialization
    Exception in thread Thread-16:
    Traceback (most recent call last):
      File "D:\Program Files\Python36\lib\threading.py", line 916, in _bootstrap_inner
        self.run()
      File "D:\Program Files\Python36\lib\threading.py", line 864, in run
        self._target(*self._args, **self._kwargs)
      File "D:\Program Files\Python36\lib\multiprocessing\pool.py", line 429, in _handle_results
        task = get()
      File "D:\Program Files\Python36\lib\multiprocessing\connection.py", line 251, in recv
        return _ForkingPickler.loads(buf.getbuffer())
    AttributeError: Can't get attribute 'Entry_XXX' on <module 'pydevd' from 'D:\\Program Files\\JetBrains\\PyCharm Community Edition 2017.1.2\\helpers\\pydev\\pydevd.py'>
    

    My code :

    from __future__ import(absolute_import, division, print_function, unicode_literals)
    
    import datetime  # For datetime objects
    import backtrader as bt
    # entries
    class Entry_XXX(bt.Indicator):
        lines = ('longsig', 'shortsig')
        params = (('XXX_param1', 5), ('XXX_param2', 10),)
    
        def __init__(self):
            self.lines.longsig = self.datas[0].close > self.datas[0].low
            self.lines.shortsig = self.datas[0].close < self.datas[0].low
            print("in entry init")
    
    # exits
    class Exit_YYY(bt.Indicator):
        lines = ('sellsig', 'coversig')
        params = (('YYY_param1', 5), ('YYY_param2', 10), ('YYY_param3', 2),)
    
        def __init__(self):
            self.lines.sellsig = self.datas[0].close > self.datas[0].low
            self.lines.coversig = self.datas[0].close > self.datas[0].low
            print(" in exit init ")
    
    # strategy
    class MasterStrategy(bt.Strategy):
    
        params = (('entry', None), ('exit', None), ('support', None),
                  ('plot_entry', True), ('plot_exit', True), ('plot_support', True),)
    
        def __init__(self):
            self.longsig = None
            self.shortsig = None
            self.sellsig = None
            self.coversig = None
    
            if self.p.entry:
                self.entry = self.p.entry(plot=self.p.plot_entry)
                self.longsig = self.entry.lines.longsig
                self.shortsig = self.entry.lines.shortsig
    
            if self.p.exit:
                self.exit = self.p.exit(plot=self.p.plot_exit)
                self.sellsig = self.exit.lines.sellsig
                self.coversig = self.exit.lines.coversig
    
            print("in the initialization")
    
        def next(self):
            # process buy, short, sell and cover signals
            pass
    
    #main code
    if __name__ == '__main__':
        cerebro = bt.Cerebro()
        strats = cerebro.optstrategy(MasterStrategy,
            entry = (Entry_XXX,),
            exit = (Exit_YYY,)
            )
        # Load data
        data = bt.feeds.GenericCSVData(
            dataname='../v#_m15_processed.csv',
            dtformat='%Y-%m-%d', fromdate=datetime.datetime(2014,1,1), todate=datetime.datetime(2015,1,1),
            datetime=0, time=1, open=2, high=3, low=4, close=5, volume=6, openinterest=8, timeframe=bt.TimeFrame.Minutes)
        cerebro.adddata(data)    
        cerebro.run()
    

  • administrators

    @asuralm said in Optimzation question:

    pydevd

    AttributeError: Can't get attribute 'Entry_XXX' on <module 'pydevd'
    

    vs

    if __name__ == '__main__':
    

    It would seem your environment is in play.

    Execute it from the console.



  • @backtrader
    Thanks it works fine in the shell.

    It's a shame not working well in pycharm.


  • administrators

    Environments like PyCharm, Spyder, Jupyter and possibly others don't work with the multiprocessing module, because they hijack the start of the Python Kernel, which prevents the proper initialization (under Windows mostly)

    This is described (for example) in:



  • @backtrader
    thanks.

    I just find a way to bypass the error. Simple enforce the strategy optimization to single thread as:

    cerebro.optstrategy(MasterStrategy, entry=entry, exit=exit, para1=[10,20])
    cerebro.run(maxcpus=1)
    

    Then IDE works with no error. At least I can debug in IDE and when I can switch to multi-threading and run in IDLE after testing.



  • @backtrader
    There is a question about the optimization improvement. I've read the article optimization-improvements, but it seems that the improvement works in the scenario of single strategy optimization.

    In this strategy auto-generation case, the problem is that trading data are the same and strategy is switching. Is this possible that the data file is only read-once and switching between strategies only?

    My current structure is like:

        signals = itertools.product(EntryList, ExitList)
        for sig, exit in signals:
            cerebro = bt.Cerebro()
            fromdate = datetime.datetime(2014, 1, 1)
            todate = datetime.datetime(2015, 1, 1)
            data = bt.feeds.GenericCSVData(...)
            cerebro.adddata(data)
    
            cerebro.optstrategy(MasterStrategy, signal=sig, exsig1=exit, [paras]...)
            cerebro.run(maxcpus=1)
    

    In such structure, the data is loaded every time in the loop and file reading is very time-consuming.
    It seems not working if I simply move the code before adddata out of the for loop.
    Is it possible that the file is read only once?

    Have you got any good suggestions to improve the performance please?

    Thanks


  • administrators

    Cerebro doesn't know that you are looping and cannot know it. The runs inside the optimization re-use the file which has already been loaded, but the runs doesn't know that you had another set of runs before with the same file.

    You could try to keep the data feed outside of the loop and execute a data.home() after the cerebro.run(...) is done to reset the pointer.

    Completely untested and may absolutely not work.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.