Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Overcoming memory limitations in optimization

    General Code/Help
    2
    2
    191
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      flair_echelon last edited by

      A strategy I'm testing has 15 variables which is proving to be difficult for my machine to handle (8 cores, 16GB RAM). Even when just using 1/2 month of hourly data causes my IDE and python to crash from lack of memory. This is with bt.Cerebro(stdstats=False). Regardless of what I set optreturn and optdatas to, it still crashes out.

      Is there a way to save data to the harddrive maybe and then run the optimization loops against that cached copy? I think a feature like that could be helpful in cutting down on the preload required for future runs on the same data.

      1 Reply Last reply Reply Quote 1
      • hghhgghdf dfdf
        hghhgghdf dfdf last edited by

        in the run method from cerebro you can intercept the variable product and choose to run only a certain number of optimizations.

        lststrats = list(itertools.product(*self.strats))
        random.shuffle(lststrats)
        lststrats = lststrats[:100]
        iterstrats = iter(lststrats)
        

        Using 5 years of hourly data I can run roughly 80-160 sessions at a time. (depending on how much memory observers/indicators take up)

        Alternatively you could roughly line up your tests with that number in mind: test in batches of 3 parameters with 6-8 variables for each parameter.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post
        Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors