Failover / Recoverability of a Backtrader strategy
During the last days I wondered: How do you continue a Backtrader strategy when it got interrupted? Let's say someone pulled the plug from your computer to put the vacuum cleaner in, or (more seriously ;-) ) you need to restart it for system updates or it simply crashed.
As there is at least some money on the line I can imagine one would not like to wake up or come home from work just to see that the computer crashed before a dump in the market and one is still in a position while the Backtrader strategy one developed for that could not run. Or the computer crashed before a run in the market where everyone tells how much of a good trader he is ;-) and one is not in a position as one would have been when the strategy would have had run.
From what I understand there is at least one thing that prevents one to safely automatically restart Backtrader with the strategy: The strategy does not know if one is in a position or not.
The feed would deliver the candle data as usual and most indicators could safely be prefilled to continue the strategy as if it would not have been stopped.
For the Parabolic SAR for example I don't think one can know in advance how many candles need to be prefilled to continue the strategy as if there was no interruption as one doesn't know when the last trend reversed and thus one can't calculate anything about the current SAR. The only solution here would be to prefill pessimistically quite a lot of candles to make sure one catches at least one trend reversal.
Depending on the strategy there can certainly be more internal state that cannot be automatically recovered and I might even overlook some obvious ones.
It is clear that there cannot be some kind of transactions over actual orders. But most of the time the strategy probably doesn't run and I would assume a crash happens at that time or otherwise accept the risk it doesn't. It would be good to at least have some kind of graceful shutdown/restart functionality to automatically recover the important transient state from a crash. I personally see only two options:
- Whitelist-based store/restore
Save only specific important data at the end of every
next()cycle and load it at the start of Backtrader if data can be found (means it run already). Only specific information e.g. if one is in a position or not would be stored and restored.
The disadvantages are:
a) that there is no hook/callback I'm aware of that signals the end of a
next()cycle when all
notify_order(),... methods have been run.
b) one might want to know if an order has been completed in the mean time. But the list of orders would have been gone after a restart. This is why one would need to store and restore internal Backtrader objects like orders which raises questions like what order properties need to be saved, what references within an order are also needed. This might even practically work with orders but it will quickly become ugly and unmanageble if more Backtrader internal state needs to be saved.
- Pickling / Unpickling the whole Cerebro object with all its references
This way at least the state will be consistant. But the disadvantage is, that one cannot easily recover the position in the runtime execution stack where the last strategy ended.
runstrategies()on the unpickled Cerebro object would reinitialize and thus overwrite some state. Calling
_runnext(runstrats)directly would probably miss some (if not all) initializations like timers if they are needed.
To solve that one would need to create something like a
continue_strategy()method that expects to be run on an unpickled Cerebro object and does only the necessary initializations. Ideally that would not be too hard as I guess the optimization feature already makes strategies run independently. But unfortunately I don't understand the Backtrader optimization code well enough to utilize the optimization feature for that. I would probably copy & paste some code from
runstrategies()in to a
continue_strategy()method, throw tests at it and go with that. But without further advise this would still be ugly.
Any advise or hints are much appreciated, even if it covers just some aspects!
Thanks in advance!