Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    How to get only end of day data when live trading

    General Code/Help
    2
    7
    2807
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jaikumarm last edited by

      I have simple test strategy based on the IB live trading blog post, it works as expected in back test but in live trading for the current day, it returns a bar as of now and does not wait till eod to give a complete bar

      0001, 2016-12-01T18:00:00.000000, 2185.5, 2191.75, 2179.25, 2186.75, 9369.0, 1, nan
      0002, 2016-12-04T18:00:00.000000, 2179.0, 2203.75, 2174.25, 2198.875, 23728.0, 1, nan
      0003, 2016-12-05T18:00:00.000000, 2199.75, 2206.5, 2195.0, 2204.75, 39177.0, 1, nan
      0004, 2016-12-06T18:00:00.000000, 2205.75, 2235.75, 2201.5, 2231.375, 111509.0, 1, nan
      0005, 2016-12-07T18:00:00.000000, 2231.25, 2246.0, 2227.75, 2242.375, 392866.0, 1, nan
      0006, 2016-12-08T18:00:00.000000, 2241.5, 2255.75, 2239.75, 2254.75, 1013273.0, 1, nan
      0007, 2016-12-11T18:00:00.000000, 2255.25, 2264.75, 2246.75, 2250.375, 1287827.0, 1, nan
      0008, 2016-12-12T18:00:00.000000, 2250.25, 2273.0, 2247.75, 2267.75, 1556179.0, 1, nan
      0009, 2016-12-13T18:00:00.000000, 2268.0, 2272.5, 2243.0, 2252.0, 1794750.0, 1, nan
      0010, 2016-12-14T18:00:00.000000, 2251.75, 2267.75, 2247.25, 2258.375, 1703803.0, 1, 0.0
      0011, 2016-12-15T18:00:00.000000, 2259.5, 2264.0, 2249.5, 2255.25, 1378336.0, 1, 1.0
      0012, 2016-12-18T18:00:00.000000, 2256.0, 2263.75, 2254.0, 2260.125, 920006.0, 1, 1.0
      0013, 2016-12-19T18:00:00.000000, 2260.25, 2269.5, 2260.0, 2266.5, 806149.0, 1, 1.0
      0014, 2016-12-20T18:00:00.000000, 2267.25, 2268.0, 2258.0, 2260.375, 648460.0, 1, 1.0
      0015, 2016-12-21T18:00:00.000000, 2258.25, 2261.0, 2251.5, 2258.75, 831630.0, 1, 0.0
      0016, 2016-12-22T18:00:00.000000, 2257.5, 2261.0, 2254.25, 2260.0, 391441.0, 1, 0.0
      0017, 2016-12-26T18:00:00.000000, 2258.75, 2269.5, 2257.75, 2260.875, 489710.0, 1, 0.0
      0018, 2016-12-27T18:00:00.000000, 2261.25, 2267.5, 2243.5, 2245.25, 974692.0, 1, 1.0
      0019, 2016-12-28T18:00:00.000000, 2245.5, 2250.0, 2239.5, 2245.125, 882916.0, 1, 0.0
      0020, 2016-12-29T18:00:00.000000, 2246.25, 2252.75, 2228.0, 2236.25, 1251896.0, 1, 1.0
      0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 0.0
      0022, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 1, 1.0
      0023, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 1, 1.0
      0024, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 1, 0.0
      0025, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 1, 0.0
      0026, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 1, 0.0
      0027, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 1, 0.0
      0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.5, 1745819.0, 1, 1.0
      ***** 2017-01-13 04:09:06.208352 DATA NOTIF: LIVE
      0029, 2017-01-12T18:00:00.000000, 2264.5, 2267.75, 2264.25, 2266.0, 68933.0, 1, 1.0
      

      I kind of have tried various combinations of replay and different timeframes and rtbar settings, not able to find the right one.

      the strategy idea is to get bar on close, then calculate a bunch of indicators and then use that "forecast" next close, based on the forecast place an market order that should fill at open and rinse and repeat every day flipping between long/short

      def runstrategy():
      
          cerebro = bt.Cerebro()
      
          storekwargs = dict(
              host='127.0.0.1', port=4003,
              clientId=110,
              timeoffset=not None,
              reconnect=3,
              timeout=3.0,
              notifyall=None,
              _debug=None
          )
      
          ibstore = bt.stores.IBStore(**storekwargs)
          broker = ibstore.getbroker()
          cerebro.setbroker(broker)
      
          timeframe = bt.TimeFrame.Days
          datacomp = 1
      
          IBDataFactory = ibstore.getdata
      
          datakwargs = dict(
              timeframe=timeframe,
              compression=datacomp,
              historical=None,
              fromdate=datetime.datetime(2016, 12, 1),
              rtbar=timeframe,
              qcheck=0.5,
              what=None,
              backfill_start=not None,
              backfill=not None,
              latethrough=None,
              tz=None
          )
      
          rekwargs = dict(
              timeframe=timeframe,
              compression=datacomp,
              bar2edge=not None,
              adjbartime=not None,
              rightedge=not None,
              takelate=not None,
          )
      
          cerebro.resampledata(dataname=data0, **rekwargs)
      
          cerebro.addstrategy(LongShortStrategyESDaily,
                              smaperiod=10,
                              trade=True,
                              exectype=bt.bt.Order.Market,
                              stopafter=None,
                              valid=valid,
                              cancel=0,
                              donotsell=None
                             )
      
          cerebro.addsizer(FixedReverser)
      
          cerebro.run()
      
      
      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        It seems your problem is:

        it returns a bar as of now and does not wait till eod to give a complete bar

        Where I guess with as of now you mean, right after delivering the last historical data bar.

        But there isn't enough information to really understand where the problem is. See for example:

        IBDataFactory = ibstore.getdata
        

        But there is no single istance of IBDataFactory in your sample above. The only reference of something being done with the data:

        cerebro.resampledata(dataname=data0, **rekwargs)
        

        Where data0 has magically appeared ...

        Addtionally from your log:

        0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.5, 1745819.0, 1, 1.0
        ***** 2017-01-13 04:09:06.208352 DATA NOTIF: LIVE
        0029, 2017-01-12T18:00:00.000000, 2264.5, 2267.75, 2264.25, 2266.0, 68933.0, 1, 1.0
        

        The last historical data is the day 2017-01-11 and the 1st day returned live seems to be 2017-01-12 which is 1 day later. It is unclear what your timezone is but this is apparently being printed out at:

        You could add details, like your own timezone, the timezone of the asset, what the localtime is when the 1st live data bar is being printed out.

        1 Reply Last reply Reply Quote 0
        • J
          jaikumarm last edited by

          thanks for the quick response. Sorry in my over-eagerness to clear our some comments in the code when I was composing the posting I deleted the IBDataFactory line.

           data0 = IBDataFactory(dataname='ES-201703-GLOBEX', **datakwargs)
          

          So to answer your queries, Am running this on a desktop in PST, and the asset is ES so its set to EST and I ran the script at 4:09AM PST on 2017-01-13.

          An you are right the last historical bar was 2017-01-11 and first live bar was 2017-01-12, but this day was not complete yet, as in the market is still open, so what I got is a valid bar at that point of time with the OHLCV calculated up that point in time. my expectation or goal is to make get the live bar return only when the day is closed.

          Here is the whole script
          https://gist.github.com/jaikumarm/b9b8124d880d683104971e25a4ab72c2

          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            If your own zone is PST then you are 3 hours behind EST (please correct this if wrong, although a quick search seems to confirm it)

            See the log

            0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.5, 1745819.0, 1, 1.0
            ***** 2017-01-13 04:09:06.208352 DATA NOTIF: LIVE
            0029, 2017-01-12T18:00:00.000000, 2264.5, 2267.75, 2264.25, 2266.0, 68933.0, 1, 1.0
            

            You get a LIVE notification at 2017-01-13 04:09:06 PST, because that timestamp is being printed in your local time. From the code in your gist at line 77:

                def notify_data(self, data, status, *args, **kwargs):
                    print('*' * 5, datetime.datetime.now(), 'DATA NOTIF:', data._getstatusname(status), *args)
                    if status == data.LIVE:
                        self.counttostop = self.p.stopafter
                        self.datastatus = 1
            

            Translated to EST the actual timestamp was: 2017-01-13 07:09:06 PST. That means that the EST has been trading already some hours into Jan 13th (over 7 hours at least)

            Data point 0029 has a timestamp with date 2017-01-12, i.e.: from the previous day, which theoretically is already over.

            The next complete 1-day bar (to be delivered when the resampling is over) should be that with date 2017-01-13.

            Unless the reasoning is very wrong, you are not (yet) getting the las bar which is still being resampled.

            1 Reply Last reply Reply Quote 0
            • J
              jaikumarm last edited by

              I tried this again today and I see an issue which might explain what is going on.

              Timezone from ContractDetails: America/Belize
              Datetime, Open, High, Low, Close, Volume, OpenPositionSize, Forecast
              ***** 2017-01-16 01:08:41.980178 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
              ***** 2017-01-16 01:08:41.980252 STORE NOTIF: <error id=-1, errorCode=2107, errorMsg=HMDS data farm connection is inactive but should be available upon demand.ilhmds>
              ***** 2017-01-16 01:08:41.980302 DATA NOTIF: DELAYED
              ***** 2017-01-16 01:08:42.480597 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
              0001, 2016-12-01T18:00:00.000000, 2185.5, 2191.75, 2179.25, 2186.75, 9369.0, 1, nan
              0002, 2016-12-04T18:00:00.000000, 2179.0, 2203.75, 2174.25, 2198.875, 23728.0, 1, nan
              0003, 2016-12-05T18:00:00.000000, 2199.75, 2206.5, 2195.0, 2204.75, 39177.0, 1, nan
              0004, 2016-12-06T18:00:00.000000, 2205.75, 2235.75, 2201.5, 2231.375, 111509.0, 1, nan
              0005, 2016-12-07T18:00:00.000000, 2231.25, 2246.0, 2227.75, 2242.375, 392866.0, 1, nan
              0006, 2016-12-08T18:00:00.000000, 2241.5, 2255.75, 2239.75, 2254.75, 1013273.0, 1, nan
              0007, 2016-12-11T18:00:00.000000, 2255.25, 2264.75, 2246.75, 2250.375, 1287827.0, 1, nan
              0008, 2016-12-12T18:00:00.000000, 2250.25, 2273.0, 2247.75, 2267.75, 1556179.0, 1, nan
              0009, 2016-12-13T18:00:00.000000, 2268.0, 2272.5, 2243.0, 2252.0, 1794750.0, 1, nan
              0010, 2016-12-14T18:00:00.000000, 2251.75, 2267.75, 2247.25, 2258.375, 1703803.0, 1, 0.0
              0011, 2016-12-15T18:00:00.000000, 2259.5, 2264.0, 2249.5, 2255.25, 1378336.0, 1, 1.0
              0012, 2016-12-18T18:00:00.000000, 2256.0, 2263.75, 2254.0, 2260.125, 920006.0, 1, 1.0
              0013, 2016-12-19T18:00:00.000000, 2260.25, 2269.5, 2260.0, 2266.5, 806149.0, 1, 1.0
              0014, 2016-12-20T18:00:00.000000, 2267.25, 2268.0, 2258.0, 2260.375, 648460.0, 1, 1.0
              0015, 2016-12-21T18:00:00.000000, 2258.25, 2261.0, 2251.5, 2258.75, 831630.0, 1, 0.0
              0016, 2016-12-22T18:00:00.000000, 2257.5, 2261.0, 2254.25, 2260.0, 391441.0, 1, 1.0
              0017, 2016-12-26T18:00:00.000000, 2258.75, 2269.5, 2257.75, 2260.875, 489710.0, 1, 1.0
              0018, 2016-12-27T18:00:00.000000, 2261.25, 2267.5, 2243.5, 2245.25, 974692.0, 1, 1.0
              0019, 2016-12-28T18:00:00.000000, 2245.5, 2250.0, 2239.5, 2245.125, 882916.0, 1, 0.0
              0020, 2016-12-29T18:00:00.000000, 2246.25, 2252.75, 2228.0, 2236.25, 1251896.0, 1, 1.0
              0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 1.0
              0022, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 1, 0.0
              0023, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 1, 1.0
              0024, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 1, 1.0
              0025, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 1, 0.0
              0026, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 1, 0.0
              0027, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 1, 0.0
              0028, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 1, 1.0
              0029, 2017-01-12T18:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 1, 1.0
              ***** 2017-01-16 01:08:43.937119 DATA NOTIF: LIVE
              0030, 2017-01-15T18:00:00.000000, 2270.25, 2270.5, 2265.0, 2266.25, 42215.0, 1, 1.0
              

              first clue of the issue is the following bar, its dated 2017-01-02 that's the CME/GLOBEX holiday
              http://www.cmegroup.com/tools-information/holiday-calendar/files/2016-holiday-calendars.zip

              0021, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 1, 1.0
              

              And from the pricing and volume numbers it looks like it actually the data from 2017-01-03, and then each date seems a day behind. which seems to make sense also the last bar 0029 as well as that was for 2017-01-13 which was last Friday.
              if not for this logic we are essentially missing data for the all the Friday's but have data for Sunday's.

              I got this from the IB gateway logs and it looks like IB data dumps are dated correctly.

              20161230-2246.25-2252.75-2228.00-2236.25-1251896-2237.80-false-124726-20170103-2240.75-2259.50-2239.50-2252.625-1786475-2248.3125-false-157661-20170104-2252.75-2267.25-2251.00-2264.25-1383745-2262.2875-false-130558-20170105-2264.50-2266.00-2254.00-2264.25-1307080-2260.825-false-131039-20170106-2264.25-2277.00-2258.25-2271.50-1541103-2269.425-false-133538-20170109-2271.25-2275.25-2263.50-2265.00-1019553-2266.80-false-103071-20170110-2264.50-2274.00-2259.50-2263.75-1299832-2266.1375-false-120978-20170111-2263.75-2271.75-2255.00-2270.625-1727931-2264.00-false-173838-20170112-2270.50-2270.50-2248.50-2263.375-1745819-2259.0375-false-166470-20170113-2264.50-2273.50-2262.75-2272.50-1182909-2269.725-false-112790-20170116-2270.25-2270.50-2265.00-2266.25-42215-2267.25-false-7503-
              

              So it looks like there is two issues, the one day roll-back for data for this instrument, and my original issue for bar 0030 2017-01-15 which is today's data which is not close yet, as you mentioned the market has been open for the last few hours but has not closed yet, so the daily bar should not be delivered yet.

              hope you can shed some light on this, also let me know if you need me to add any further debug logs or print statements that might help.

              1 Reply Last reply Reply Quote 0
              • J
                jaikumarm last edited by jaikumarm

                wasn't sure if I made some errors with my modifications to the ibtest.py sample. here is the run from ibtest.py unmodified from master branch

                python ibtest.py --port 4003 --data0 ES-201703-GLOBEX --resample --timeframe Days --compression 1 --fromdate 2017-01-01
                Server Version: 76
                TWS Time at connection:20170116 10:12:19 UTC
                --------------------------------------------------
                Strategy Created
                --------------------------------------------------
                Timezone from ContractDetails: America/Belize
                Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
                ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
                ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
                ***** DATA NOTIF: DELAYED
                Data0, 0001, 736332.0, 2017-01-02T18:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 0, nan
                Data0, 0002, 736333.0, 2017-01-03T18:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 0, nan
                Data0, 0003, 736334.0, 2017-01-04T18:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 0, nan
                Data0, 0004, 736335.0, 2017-01-05T18:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 0, nan
                Data0, 0005, 736338.0, 2017-01-08T18:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 0, 2263.525
                Data0, 0006, 736339.0, 2017-01-09T18:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 0, 2265.75
                Data0, 0007, 736340.0, 2017-01-10T18:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 0, 2267.025
                Data0, 0008, 736341.0, 2017-01-11T18:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 0, 2266.85
                Data0, 0009, 736342.0, 2017-01-12T18:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 0, 2267.05
                ***** DATA NOTIF: LIVE
                Data0, 0010, 736345.0, 2017-01-15T18:00:00.000000, 2270.25, 2270.5, 2265.0, 2267.0, 49079.0, 0, 2267.45
                

                One additional data point am running the IB gateway on a docker container which is set to UTC by default.

                am going to build a new container which is set to EST and run everything inside that and see if any of these TZ differences are making an impact. thanks for your help so far in troubleshooting this.

                1 Reply Last reply Reply Quote 0
                • J
                  jaikumarm last edited by

                  Okay looks like I can't read instructions :(.. I tried setting the timezone in the ibtest.py script to PST and voila! all the problems solved. all the datetime stamps are aligned and the last daily bar is waiting for the day to complete, I hope!

                  python ibtest.py --port 4003 --data0 ES-201703-GLOBEX --resample --timeframe Days --compression 1 --fromdate 2017-01-01 --timezone PST
                  Server Version: 76
                  TWS Time at connection:20170116 10:34:35 UTC
                  --------------------------------------------------
                  Strategy Created
                  --------------------------------------------------
                  Timezone from ContractDetails: America/Belize
                  Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
                  ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:usfuture>
                  ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:ilhmds>
                  ***** DATA NOTIF: DELAYED
                  Data0, 0001, 736332.0, 2017-01-03T00:00:00.000000, 2240.75, 2259.5, 2239.5, 2252.625, 1786475.0, 0, nan
                  Data0, 0002, 736333.0, 2017-01-04T00:00:00.000000, 2252.75, 2267.25, 2251.0, 2264.25, 1383745.0, 0, nan
                  Data0, 0003, 736334.0, 2017-01-05T00:00:00.000000, 2264.5, 2266.0, 2254.0, 2264.25, 1307080.0, 0, nan
                  Data0, 0004, 736335.0, 2017-01-06T00:00:00.000000, 2264.25, 2277.0, 2258.25, 2271.5, 1541103.0, 0, nan
                  Data0, 0005, 736338.0, 2017-01-09T00:00:00.000000, 2271.25, 2275.25, 2263.5, 2265.0, 1019553.0, 0, 2263.525
                  Data0, 0006, 736339.0, 2017-01-10T00:00:00.000000, 2264.5, 2274.0, 2259.5, 2263.75, 1299832.0, 0, 2265.75
                  Data0, 0007, 736340.0, 2017-01-11T00:00:00.000000, 2263.75, 2271.75, 2255.0, 2270.625, 1727931.0, 0, 2267.025
                  Data0, 0008, 736341.0, 2017-01-12T00:00:00.000000, 2270.5, 2270.5, 2248.5, 2263.375, 1745819.0, 0, 2266.85
                  Data0, 0009, 736342.0, 2017-01-13T00:00:00.000000, 2264.5, 2273.5, 2262.75, 2272.5, 1182909.0, 0, 2267.05
                  ***** DATA NOTIF: LIVE
                  

                  PS: I fixed it in my modified script and it works there as well. thanks Daniel for your help and for backtrader!!

                  1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post
                  Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors