Optimizing with IBStore causes redundant connections/downloads

d416

I'm trying to optimize a Simple Moving Average cross-over strategy using Interactive Brokers data via IBStore.

While optimizing, the processes initiate numerous connects/disconnects to the IB Trader Workstation as the parameters are iterated - that is, one connection for each permutation of parameters. Both the TWS data window shows this, and a similar number of "TWS Time at connection:20170514 16:06:03 EST" lines are printed on the console. (If the optimization is iterating through 100 permutations, there are 100 connections)

I'm guessing that with each iteration, no matter what the data source, the data is loaded into the strategy? I figured the default values for preload/runonce/optdatas would handle this.

If this is not the case, is the proper way to do this just pre-load the required data set into Pandas or influxdb and use it from there?

from __future__ import (absolute_import, division, print_function,
                    unicode_literals)

import datetime  # For datetime objects
import os.path  # To manage paths
import sys  # To find out the script name (in argv[0])

# Import the backtrader platform
import backtrader as bt

# Create a Data Feed
ibstore = bt.stores.IBStore(host='127.0.0.1',port=7496,clientId=5)
mysymbol = 'EUR.USD-CASH-IDEALPRO'
fromdate = datetime.datetime(2015,1,1,00,00)
todate = fromdate = datetime.datetime(2017,3,30,00,00)
data = ibstore.getdata(dataname=mysymbol, timeframe=bt.TimeFrame.Days, compression=1, fromdate=fromdate, todate=todate, historical=True)


# Create a Strategy
class SmaCross(bt.SignalStrategy):
params = (
    ('sma1', 10),
    ('sma2', 30),
)
def __init__(self):
    SMA1 = bt.ind.SMA(period=int(self.params.sma1))
    SMA2 = bt.ind.SMA(period=int(self.params.sma2))
    crossover = bt.ind.CrossOver(SMA1, SMA2)
    self.signal_add(bt.SIGNAL_LONG, crossover)



if __name__ == '__main__':
# Create a cerebro entity
cerebro = bt.Cerebro(maxcpus=1)

# Add a strategy
strats = cerebro.optstrategy(
    SmaCross,
    sma1=range(50, 60),
    sma2=range(50,60))

# Add the Data Feed to Cerebro
cerebro.adddata(data)

# Set our desired cash start
cerebro.broker.setcash(10000.0)

# Add a FixedSize sizer according to the stake
cerebro.addsizer(bt.sizers.FixedSize, stake=1000)

# Set the commission
cerebro.broker.setcommission(commission=0.0)

# Run over everything
cerebro.run()

backtrader

Optimization was in the platform long before any kind of live streams was added and will simply ask the streams to load its data if the data is not preloaded.

In this case the data is not seen as preloaded because of the live nature of the feed and that's why it is being recollected.

It was never envisioned (also taking into account the historical download limitations inherent to IB) that such feeds would be used in an optimization attempt.

d416

@backtrader Than you for the confirm (and this gift of software!)

It seems the answer then is to

Get a snapshot of IB historical data and load into a more optimal store, such as a Pandas data frame feed
Use the Pandas data frame as a data feed in the optimization

On point 1, is there an easy way you can think of to copy the ibstore data into a Pandas data frame? Or perhaps this would best be handled using the external IBpy methods to prep the data.

-D

d416

Resolved: see this post for a solution...

https://community.backtrader.com/topic/447/code-snippet-copying-ib-data-to-pandas-for-efficiency

backtrader

The message above got under the radar. A potential solution (untested) would be:

class MyData(bt.feeds.IBData):
    def islive(self):
        '''Usually Returns ``True`` to notify ``Cerebro`` that preloading and runonce
        should be deactivated

        --- Returns False in subclass to allow optimization
        '''
        return False

and then

bt.stores.IBStore.DataCls = MyData