Example of adding live-data to static in strategy?

RandyT

Trying to get my head around the following set of requirements. I'm taking the system that I have coded that is succesfully backtesting against static data and now trying to add in a live data source to do more realtime calculation of signals.

Data0 = Live data for traded instrument 'BAR'
Data1 = Live data for signaling instrument 'FOO' used to signal entry/exit for the current day
Data2 =  Years of static daily timeframe data for signaling instrument 'FOO'

I need to add the close value for probably a minute timeframe off of Data1 to the value from yesterday and beyond found in daily timeframe data of Data2.

I've looked at most of the samples and am not finding anything that is augmenting the Data2 to calculate the indicator values that need lookback of multiple years.

This all works when dealing with daily static data provided by Data2. I'm just a bit confused as to where to take the value of the minute close of Data1 as if it was the Daily close value for today. I am really only interested in triggering a signal off of the actual close for today to execute a trade on the next bar of Data0.

As a side note, I don't really trust the IB historical data enough to use it for the multi-year lookback needed here, so I use a different static source for the historical daily data and really only want to add today's values to that other static data source to calculate the signal.

Advance thanks for your help.

backtrader

The topic seems to summarize itself as: *How to calculate an indicator which is based in data from different timeframes? *

This is supported through a mechanism that was named: Coupling

See here:

https://www.backtrader.com/docu/mixing-timeframes/indicators-mixing-timeframes.html

or here

https://www.backtrader.com/blog/posts/2016-05-05-indicators-mixing-timeframes/indicators-mixing-timeframes.html

(The content should be basically the same)

Note: the mechanism works but cross-plotting doesn't. The changes involved in releasing the 1.9.x series for a new data synchronization mechanism, meant changes in plotting and the mechanism originally used for the cross-plotting no longer works.

RandyT

Still struggling with this concept a bit.

How do I specifically couple data2 and data1 from my example above?

Also, looking at the example code you referenced, does the resample implicitly create data1?

I'm doing something like the following:

    # Parse static ES data file
    data0 = bt.feeds.PandasData()
    cerebro.adddata(data0, name='ES')

    # Parse SPY data file
    data1 = bt.feeds.PandasData()
    cerebro.adddata(data1, name='SPY')

if live:
        IBDataFactory = ibstore.getdata
        data2 = IBDataFactory(dataname=symbol)
        cerebro.adddata(data2, name='ES-live')

        # SPY Live Data
        data3 = IBDataFactory(dataname=symbol)
        cerebro.adddata(data3, name='SPY-live')

I've obviously left out a lot of the details in that code but the connection works and am getting data based on the log info of the IBgateway.

I'm still not clear on how I couple data1 and data3 to get live SPY data to pass to my indicators. I've spent a lot of time in the debugger trying to sort this out and am not finding the magic.

Thanks for your help.

RandyT

@backtrader I also wonder if we are not talking about different issues here.

In Live data mode, I have 4 datas. 2 of those datas sources are static data read from file in order to give me enough backdata to feed my indicators. The other 2 datas are live streaming data that I wish to execute trades against one of them and to continue to calculate the indicator values based on the new live data coming in to provide today's values for the static data from yesterday and beyond.

I am not resampling to generate this data.

backtrader

This was completely misunderstood. It seems you are simply looking for backfilling to warm up the indicators.

The IBData supportf backfilling from existing data sources with:

backfill_from (default: None)

An additional data source can be passed to do an initial layer of backfilling. Once the data source is depleted and if requested, backfilling from IB will take place. This is ideally meant to backfill from already stored sources like a file on disk, but not limited to.

    # Parse static ES data file
    data0 = bt.feeds.PandasData()

    # Parse SPY data file
    data1 = bt.feeds.PandasData()

    IBDataFactory = ibstore.getdata
    data2 = IBDataFactory(dataname=symbol)
    cerebro.adddata(data2, name='ES-live', backfill_from=data0)

    # SPY Live Data
    data3 = IBDataFactory(dataname=symbol, backfill_from=data1)
    cerebro.adddata(data3, name='SPY-live')

RandyT

Excellent. I will give this a try.

As I wrote above, I do not trust the IB historical data. I have another source of very solid historical data that I prefer to use and would then like to augment that with the current day's data for making trading decisions.

backtrader

That's the point of passing your data0 which you have loaded from a file as backfill_from. Anything which is in that file will be considered, and only the delta between the end of the file and the start of the live feed will be pulled from IB.

RandyT

Just a bit of a clarification for anyone running across this thread in the future, the backfill_from parameter should be used with the call to IBDataFactory() instead of the .adddata() object as show in one of the examples above.

That said, I'm running into the following error when I attempt to take this approach:

  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 809, in run
    runstrat = self.runstrategies(iterstrat)
  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 933, in runstrategies
    self._runnext(runstrats)
  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 1152, in _runnext
    drets = [d.next(ticks=False) for d in datas]
  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 1152, in <listcomp>
    drets = [d.next(ticks=False) for d in datas]
  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/feed.py", line 339, in next
    ret = self.load()
  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/feed.py", line 411, in load
    _loadret = self._load()
  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/feeds/ibdata.py", line 528, in _load
    for alias in self.lines.getaliases():
AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'getaliases'

backtrader

Indeed. That's the problem with snippet typing ... one of the backfill_from got wrongly positioned (luckily the 2nd was where it had to be) The corrected snippet.

  # Parse static ES data file
    data0 = bt.feeds.PandasData(dataname='my-es-file')

    # Parse SPY data file
    data1 = bt.feeds.PandasData(dataname='my-spy-file')

    IBDataFactory = ibstore.getdata
    data2 = IBDataFactory(dataname=symbol, backfill_from=data0)
    cerebro.adddata(data2, name='ES-live')

    # SPY Live Data
    data3 = IBDataFactory(dataname=symbol, backfill_from=data1)
    cerebro.adddata(data3, name='SPY-live')

The getaliases seems to be one of the points where a refactoring went wrong. The correct call is getlinealiases and has been pushed to the master.

RandyT

With this change, when running in live mode now with the backfill_from , I don't seem to have an initialized self.position in next() of strategy.

Not sure yet if this is something that I have failed to do or issue.

This works when not running with live data with local static data only.

Looking at this in the debugger:

(Pdb) foo = self.broker.getposition(data=self.data_es)
(Pdb) foo.size
0

So the data is there, just not initialized to self.position.

backtrader

The status/size reported by self.position (which is a Position instance with size and price attributes) is not related to what's being used to fill the value in the self.data.

In your original example there are 4 data feeds and now the system should be down to 2. self.position is a shortcut to get the actual position on self.datas[0] (aka self.data or self.data0).

Additionally and if code has been used from ibtest.py, any buy/sell action will first take place when the LIVE notification is received (the initial status whilst backfilling is DELAYED

RandyT

Thanks for the hint. That seems to have resolved my issue.

Count me as one who is officially running backtrader fully auto on paper account.

@backtrader thanks again for all of your help and a great set of tools that you have made available to us all.

RandyT

Continuing to have some trouble here.

System happily waited for live data to start flowing. However, as soon as the market opened this evening, the system crashed with the following backtrace.

Server Version: 76
TWS Time at connection:20170102 19:10:03 MST
Traceback (most recent call last):
File "systems/system.py", line 335, in <module> runstrategy()
File "systems/system.py", line 149, in runstrategy
    results = cerebro.run(runonce=False, tradehistory=True, exactbars=args.exactbars)
File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 809, in run
    runstrat = self.runstrategies(iterstrat)
File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 933, in runstrategies
    self._runnext(runstrats)
File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/backtrader/cerebro.py", line 1166, in _runnext                                        
    dt0 = min((d for i, d in enumerate(dts)

ValueError: min() arg is an empty sequence

I've set a break in notify_data() as follows and am never getting there. Status is never data.LIVE.
The market data is available as I have another system that is able to pull the requested data.

def notify_data(self, data, status, *args, **kwargs):
        if self.p.printout:
            print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args)

        if status == data.LIVE:
            self.datastatus = True
            import pdb; pdb.set_trace()

Also getting the following prints:

***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:euhmds>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:fundfarm>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ushmds>

I can seemingly resolve this issue by first doing .adddata() for the live data stream before the .resample(). But I was under the impression that the .adddata() was not required if doing a .resample() as seen in the ibtest.py code.

I never get LIVE data status though with this configuration.

backtrader

The <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfarm> notifications are OK. IB chose to notify that the status of the different data server through errors 21xx. The text in errMsg is what's actually makes the diffference, but in this case it says OK.

There shouldn't be any need to use adddata, as you have already seen in the sample.

From the trace, the error can be tracked down to:

dt0 = min((d for i, d in enumerate(dts)

ValueError: min() arg is an empty sequence

The entire code fragment which is in play:

                # Get index to minimum datetime
                if onlyresample or noresample:
                    dt0 = min((d for d in dts if d is not None))
                else:
                    dt0 = min((d for i, d in enumerate(dts)
                               if d is not None and i not in rsonly))

Your code fails in the 2nd part which is for cases in which no resampling takes place or a mix of adddata and resample/replay takes place.

In any case, one can only be in that code fragment if one of the existing data feeds has reported to have something to deliver.

This is only a guess, because there is no description in your post as to what you are actually doing (why you don't get LIVE is part of the unknown configuration)

RandyT

One suspicion that I would first ask, is it possible to use backfill_from with data in different timeframe (Daily in my case) than the data timeframe used for the live feed?

To give a bit more detail as I try to sort through where this might be failing... I am doing the following:

 # Parse static ES data file of daily OHLCV
    data0 = bt.feeds.PandasData(dataname='my-es-file') 

    # Parse SPY data file of daily OHLCV
    data1 = bt.feeds.PandasData(dataname='my-spy-file')

    IBDataFactory = ibstore.getdata
    # ES Live Data
    data2 = IBDataFactory(dataname=symbol, backfill_from=data0, 
                                             timeframe=bt.Timeframe.Seconds,
                                             fromdate=None,
                                             historical=False,
                                             rtbar=False,
                                             qcheck=0.5,
                                             what=None,
                                             latethrough=False,
                                             tz=None,
                                             backfill_start=True,
                                             backfill=True,
                                              compression=5)
    cerebro.resampledata(data3, timeframe=bt.TimeFrame.Minutes, compression=1, 
                                             bar2edge=True,
                                             adjbartime=True,
                                             rightedge=True,
                                             takelate=True)

    # SPY Live Data
    data3 = IBDataFactory(dataname=symbol, backfill_from=data1,
                                             timeframe=bt.TimeFrame.Minutes,
                                             fromdate=None,
                                             historical=False,
                                             rtbar=False,
                                             qcheck=0.5,
                                             what=None,
                                             latethrough=False,
                                             tz=None,
                                             backfill_start=True,
                                             backfill=True)
    cerebro.resampledata(data3, timeframe=bt.TimeFrame.Days, compression=1, 
                                             bar2edge=True,
                                             adjbartime=True,
                                             rightedge=True,
                                             takelate=True)

With the above setup, I now error out in my indicator with the following error:

File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/back
trader/linebuffer.py", line 182, in get                                           return list(islice(self.array, start, end))
ValueError: Indices for islice() must be None or an integer: 0 <= x <= sys.max
size.

In the debugger, I see:

(Pdb) len(self)
756
(Pdb) len (self.dval8)
756
(Pdb) self.dval8[0]
0.49771632773232516
(Pdb) self.dval8.get(size=252, ago=0)
*** ValueError: Indices for islice() must be None or an integer: 0 <= x <= sys
.maxsize. 
(Pdb) self.dval8.get(size=8, ago=0)
[0.49987104505171959, 0.5011817035953996, 0.50303547679268779, 0.5033757977972
1131, 0.5014079085533496, 0.4985198983481765, 0.49832603424557598, 0.49771632773232516]

Requesting a slice bigger than 8 items fails with the above error.

backtrader

is it possible to use backfill_from with data in different timeframe (Daily in my case) than the data timeframe used for the live feed?

The platform will NOT prohibit that you do that, but the outcome is clearly undefined. Indicators (if working at all) will be calculating values based on completely different bar sizes (for example) and the outcome will make no sense.

Requesting a slice bigger than 8 items fails with the above error.

The default ibtest.py applies exactbars=1 to cerebro.run(...). Although not shown above, this is for sure the problem. This switches all objects into only buffering the minimum needed amount of data, as reported by the dependencies recognized during creation.

Your indicator has reported to only need a lookback period of 8 positions. Looking back 252 positions breaks it.

RandyT

If I run with exactbars=0, I get the error previously reported:

  File "/home/inmate/.virtualenvs/backtrader3/lib/python3.4/site-packages/back
trader/cerebro.py", line 1166, in _runnext                                        
     dt0 = min((d for i, d in enumerate(dts)
ValueError: min() arg is an empty sequence

exactbars set to 1 or -1 produce the error in the indicator.

In my indicator, I have the following in __init__():

   def __init__(self):
        self.addminperiod(756)

This is why I see a size of 756 when we hit next(): for the first time in the indicator. How can I have a size of 756 yet not have 756 values available to the .get?

To clarify the first question about different sizes, I am resampling the live data to the same timeframe as the static local data, so I am assuming that resolves any issue there. I am only using the SPY data for indicator calculations, so I would not expect that to be an issue.

backtrader

Because there is always a minimum period of 1 which is transmitted by data feeds. Try adding 757

RandyT

Adding 757 as minperiod has the same error:

(Pdb) len(self)
757

This does work with static data only. It is the addition of the live data and backfill_from that is producing this issue.

backtrader

(Correction added above to remark that "The platform will NOT ... when mixing different timeframes with backfill_from)

It may be good to try to summarize the situation. (Please correct any misinterpretation):

Above the indicator was looking to get 252 items, but it failed because it only had 8 in the buffer, because the code is working with exactbars=1 which reduces the buffers of all objects to the minimum required by the dependencies.
756 was not working in spite of having self.addminperiod(756). This was interpreted, probably wrong, as you not having the full 756 values but probably 755 due to the overlapping nature of the datas.

May it be that you still had only the 8 items from the post above?

Summary of exactbars:

        Possible values:
          - ``True`` or ``1``: all "lines" objects reduce memory usage to the
            automatically calculated minimum period.

            If a Simple Moving Average has a period of 30, the underlying data
            will have always a running buffer of 30 bars to allow the
            calculation of the Simple Moving Average

            - This setting will deactivate ``preload`` and ``runonce``
            - Using this setting also deactivates **plotting**

          - ``-1``: data feeds and indicators/operations at strategy level will
            keep all data in memory.

            For example: a ``RSI`` internally uses the indicator ``UpDay`` to
            make calculations. This subindicator will not keep all data in
            memory

            - This allows to keep ``plotting`` and ``preloading`` active.

            - ``runonce`` will be deactivated

          - ``-2``: data feeds and indicators kept as attributes of the
            strategy will keep all points in memory.

            For example: a ``RSI`` internally uses the indicator ``UpDay`` to
            make calculations. This subindicator will not keep all data in
            memory

            If in the ``__init__`` something like
            ``a = self.data.close - self.data.high`` is defined, then ``a``
            will not keep all data in memory

            - This allows to keep ``plotting`` and ``preloading`` active.