For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Restarting Backtrader for every candle



  • Hello,

    I would like to run Backtrader in a Google Cloud configuration. More precisely I would like to use the Google Cloud Scheduler to run the Backtrader strategy e.g. every minute in a Google Cloud Function.
    But that would mean running Cerebro for the first next() call, pickling the instance before the subsequent next() call and stopping the execution via self.env.runstop(). That will probably take way less than a minute.
    At the start of the next minute the Google Cloud Scheduler starts the Google Cloud Function again with the Backtrader runtime. It should then unpickle the Cerebro instance and continue to run it.

    I'm aware that Backtrader is probably not made for that scenario.

    But that way to run it in the Google Cloud would most probably still be covered by the free plan while still getting a high reliability of the infrastructure (hardware, internet,...) the code runs on.

    Pickling and unpickling the state of the Cerebro instance works like a charm (after some changes in my code) but:

    1. I don't know when exactly I should pickle the state. It should be at a time when next() has been finished and notify_data(), notify_order() and so on have been called.
    2. At the start of the next minute the Cerebro instance will get unpickled but I don't know how to rerun Cerebro with that instance. cerebro.runstrategies(cerebro.strats[0]) would probably be too early.

    Is there a way / workaround to support that scenario with the current version of Backtrader?
    Would that "restartability" feature be something that Backtrader could support in the future?

    Any help is much appreciated!

    Thanks,

    Sandro


  • administrators

    @Sandro-Böhme said in Restarting Backtrader for every candle:

    I'm aware that Backtrader is probably not made for that scenario.

    You can remove the "probably". It is not.

    @Sandro-Böhme said in Restarting Backtrader for every candle:

    Is there a way / workaround to support that scenario with the current version of Backtrader?

    No.

    @Sandro-Böhme said in Restarting Backtrader for every candle:

    Would that "restartability" feature be something that Backtrader could support in the future?

    No.



  • Just curious - why add the dependency on Google API - just because it fits into the always free plan ?

    The alternative could be to just get the N1 VM instance and use the 300$ free trial credit. Such node will cost you around 25$ monthly (assuming 10GB of storage and Linux OS - which is probably more than you need, given your trading frequency) so that this credit will cover a full year of trial.

    Then you may just run whatever you want on this node, using a standard Backtrader API without any dependency on Google Cloud APIs.


  • administrators

    @vladisld said in Restarting Backtrader for every candle:

    Just curious - why add the dependency on Google API - just because it fits into the always free plan ?

    It's the Google like Amazon Lambda. It is the current trend to have serverless services. Instead of managing a server, a script/function/piece of code is the thing to manage. The rest is managed in the background by the provider (Google, Amazon, ...)



  • @vladisld said in Restarting Backtrader for every candle:

    Just curious - why add the dependency on Google API - just because it fits into the always free plan ?

    Yea, partially because it's more cost effective. I really try to avoid monthly costs. Especially if I want to run several strategies and an N1 VM might not be enough it gets cheaper than a full VM.

    An other reason is, that I could automate or externally control the strategy creation. Assume you have a strategy with a trading pair and period parameter. You can let a Google Cloud Function listen on a Google Firestore node that contains this parameters as the trade setup. When a new one with the actual trading pair and period/timeframe got created it triggers the Google Cloud Function that listens which can then configure the Google Cloud Scheduler with the strategy that also runs in a separate Function. That strategy Function would start every minute and stop after a next() cycle has ended if a 1 minute period had been configured. The strategy can of course also write to a subnode for the trade setup and report things like current win rate, profit/loss and so on.

    This setup is what I'm working on. It is pretty easy to create/manage Firestore nodes using the Google frontend. To keep the things simple for now this is what I would go for at the beginning. But it should also not be too hard to generate a website with this Firestore nodes using a JAMstack.

    The actual development and backtesting of the strategy can still be done locally.

    I've created proof of work's for the triggers and for running Backtrader in a Google Function. That in general is not a problem. I also worked on a whitelist based persistence of the strategy where I only saved some custom strategy variables and some internal ones from Backtrader. Then I started Backtrader as usual in a Function with a custom start time based on the Cloud Scheduler and used my custom loading to overwrite some strategy variables.
    But at some point I was not sure if I really catched all the needed properties to overwrite. This is why started just pickling everything to be sure I've got the whole state from the last run. But now I have no workaround or support to restart Cerebro with that data.



  • @backtrader said in Restarting Backtrader for every candle:

    @Sandro-Böhme said in Restarting Backtrader for every candle:

    I'm aware that Backtrader is probably not made for that scenario.

    You can remove the "probably". It is not.

    @Sandro-Böhme said in Restarting Backtrader for every candle:

    Is there a way / workaround to support that scenario with the current version of Backtrader?

    No.

    @Sandro-Böhme said in Restarting Backtrader for every candle:

    Would that "restartability" feature be something that Backtrader could support in the future?

    No.

    Ok, I thought about a configuration where Backtrader is run for trading as usual but then I wondered how one safely restarts it after it has been stopped. I mean, it probably will stop at one point. But then the strategy doesn't know it's last state e.g. if it's in a position and so on.



  • @Sandro-Böhme said in Restarting Backtrader for every candle:

    Ok, I thought about a configuration where Backtrader is run for trading as usual but then I wondered how one safely restarts it after it has been stopped. I mean, it probably will stop at one point. But then the strategy doesn't know it's last state e.g. if it's in a position and so on.

    I'll create a separate post about that later.



  • @Sandro-Böhme said in Restarting Backtrader for every candle:

    @vladisld said in Restarting Backtrader for every candle:

    Just curious - why add the dependency on Google API - just because it fits into the always free plan ?

    Yea, partially because it's more cost effective. I really try to avoid monthly costs. Especially if I want to run several strategies and an N1 VM might not be enough it gets cheaper than a full VM.

    An other reason is, that I could automate or externally control the strategy creation. Assume you have a strategy with a trading pair and period parameter. You can let a Google Cloud Function listen on a Google Firestore node that contains this parameters as the trade setup. When a new one with the actual trading pair and period/timeframe got created it triggers the Google Cloud Function that listens which can then configure the Google Cloud Scheduler with the strategy that also runs in a separate Function. That strategy Function would start every minute and stop after a next() cycle has ended if a 1 minute period had been configured. The strategy can of course also write to a subnode for the trade setup and report things like current win rate, profit/loss and so on.

    This setup is what I'm working on. It is pretty easy to create/manage Firestore nodes using the Google frontend. To keep the things simple for now this is what I would go for at the beginning. But it should also not be too hard to generate a website with this Firestore nodes using a JAMstack.

    The actual development and backtesting of the strategy can still be done locally.

    I've created proof of work's for the triggers and for running Backtrader in a Google Function. That in general is not a problem. I also worked on a whitelist based persistence of the strategy where I only saved some custom strategy variables and some internal ones from Backtrader. Then I started Backtrader as usual in a Function with a custom start time based on the Cloud Scheduler and used my custom loading to overwrite some strategy variables.
    But at some point I was not sure if I really catched all the needed properties to overwrite. This is why started just pickling everything to be sure I've got the whole state from the last run. But now I have no workaround or support to restart Cerebro with that data.

    To be fair - it's probably quite complex to support recoverability.



  • @backtrader said in Restarting Backtrader for every candle:

    @vladisld said in Restarting Backtrader for every candle:

    Just curious - why add the dependency on Google API - just because it fits into the always free plan ?

    It's the Google like Amazon Lambda. It is the current trend to have serverless services. Instead of managing a server, a script/function/piece of code is the thing to manage. The rest is managed in the background by the provider (Google, Amazon, ...)

    BTW: The problem with Amazon Lambda, Google Functions and so on is, that those code usually needs a lot of dependencies with makes startup time of this code very long. Backtrader is a very very good fit in this aspect as it doesn't has this problem!


Log in to reply
 

});