Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Replay data and backfilling on real time

    General Code/Help
    2
    4
    1296
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ealvarpe
      ealvarpe last edited by

      Trying to run the ibtest script from the source code with this command

      ./ibtest.py --replay --timeframe Days --compression 1 --data0 EUR.USD-CASH-IDEALPRO
      

      and the log for the first data received onto the backfilling is:

      ***** DATA NOTIF: DELAYED
      Data0, 0001, 736088.1666666665, 2016-05-03T23:59:59.999986, 1.1531, 1.1616, 1.1495, 1.1496, -1.0, 0, nan
      Data0, 0001, 736089.1666666665, 2016-05-04T23:59:59.999986, 1.1531, 1.1616, 1.1466, 1.1486, -2.0, 0, nan
      Data0, 0002, 736090.1666666665, 2016-05-05T23:59:59.999986, 1.14865, 1.14935, 1.13855, 1.14045, -1.0, 0, nan
      Data0, 0002, 736091.1666666665, 2016-05-06T23:59:59.999986, 1.14865, 1.14935, 1.13855, 1.1406, -2.0, 0, nan
      Data0, 0003, 736094.1666666665, 2016-05-09T23:59:59.999986, 1.13945, 1.14195, 1.1375, 1.1383, -1.0, 0, nan
      Data0, 0003, 736095.1666666665, 2016-05-10T23:59:59.999986, 1.13945, 1.14195, 1.13585, 1.13705, -2.0, 0, nan
      Data0, 0004, 736096.1666666665, 2016-05-11T23:59:59.999986, 1.13735, 1.14465, 1.1369, 1.14255, -1.0, 0, nan
      

      The first thing I see wrong is that the open and high seems to be consecutive the same every 2 days.

      Also, at this point in time if I try to access the close of the data, this defers from what´s received on the IB feed:
      So If I print last data0.close is:

      datas[0].close[0] = 1.1376
      datas[0].close[-1] = 1.13705
      datas[0].close[-2] = 1.1406
      datas[0].close[-3] = 1.1486
      datas[0].close[-4] = 1.1376
      datas[0].close[-5] = 1.13705
      datas[0].close[-6] = 1.1406
      datas[0].close[-7] = 1.1486
      datas[0].close[-8] = "Not available yet"`
      

      How is the data being backfilled?

      Also, when the data reach the "LIVE" status, so the feed starts consuming in real time I don´t have access to the whole data backfilled; only the last values.

      I´ve tried also with other securities like SPY-STK-SMART-USD and the behaviour is the same.

      Any ideas?

      1 Reply Last reply Reply Quote 0
      • ealvarpe
        ealvarpe last edited by

        Seems that the access to old backhistory was related to running with exactbars=1.
        When changed, removing this property, I have access to all the history.
        But still with the data issue...

        1 Reply Last reply Reply Quote 0
        • ealvarpe
          ealvarpe last edited by

          I´ve tried to run the same script with "resample" instead of replay and the data is backfilled accordingly to what it´s received.
          But the data received seems to be losing 1 of each 2 consecutive days... (I´ve set the --debug option and I´ve verified that IB data is being received correctly)
          Am I doing something wrong or do I need to use some extra parameter?

          Data0, 0103, 736385.2083333333, 2017-02-25T00:00:00.000000, 236.27, 236.99, 235.21, 236.74, 1423468.0, 0, 234.906
          Data0, 0104, 736389.2083333333, 2017-03-01T00:00:00.000000, 236.8, 237.31, 236.01, 236.93, 1340010.0, 0, 235.998
          Data0, 0105, 736391.2083333333, 2017-03-03T00:00:00.000000, 237.42, 240.32, 237.42, 238.16, 1547475.0, 0, 236.62800000000001
          Data0, 0106, 736393.2083333333, 2017-03-05T00:00:00.000000, 237.88, 238.61, 237.66, 238.35, 689651.0, 0, 237.27800000000002
          Data0, 0107, 736397.2083333333, 2017-03-09T00:00:00.000000, 237.52, 238.12, 236.36, 236.73, 1712065.0, 0, 237.382
          Data0, 0108, 736399.2083333333, 2017-03-11T00:00:00.000000, 236.79, 238.32, 235.74, 237.78, 1464157.0, 0, 237.59
          Data0, 0109, 736403.1666666665, 2017-03-14T23:59:59.999986, 237.45, 237.89, 236.19, 237.08, 1009787.0, 0, 237.61999999999998
          Data0, 0110, 736405.1666666665, 2017-03-16T23:59:59.999986, 237.32, 239.75, 237.29, 238.6, 1390938.0, 0, 237.708
          Data0, 0111, 736407.1666666665, 2017-03-18T23:59:59.999986, 237.42, 238.23, 236.88, 236.98, 700959.0, 0, 237.43400000000003
          Data0, 0112, 736411.1666666665, 2017-03-22T23:59:59.999986, 237.06, 237.61, 233.05, 234.51, 2358886.0, 0, 236.99
          Data0, 0113, 736413.1666666665, 2017-03-24T23:59:59.999986, 234.25, 235.34, 232.96, 233.94, 1885646.0, 0, 236.22199999999998
          Data0, 0114, 736417.1666666665, 2017-03-28T23:59:59.999986, 232.1, 235.81, 231.61, 235.12, 1505616.0, 0, 235.83
          Data0, 0115, 736419.1666666665, 2017-03-30T23:59:59.999986, 235.26, 236.52, 234.72, 236.21, 990011.0, 0, 235.352
          Data0, 0116, 736421.1666666665, 2017-04-01T23:59:59.999986, 235.76, 236.51, 235.49, 235.65, 618459.0, 0, 235.086
          Data0, 0117, 736425.1666666665, 2017-04-05T23:59:59.999986, 236.13, 237.39, 233.91, 234.31, 2251815.0, 0, 235.046
          Data0, 0118, 736427.1666666665, 2017-04-07T23:59:59.999986, 234.23, 236.04, 234.15, 235.19, 1347907.0, 0, 235.296
          Data0, 0119, 736431.1666666665, 2017-04-11T23:59:59.999986, 235.26, 236.26, 233.34, 235.03, 1248691.0, 0, 235.27800000000002
          Data0, 0120, 736433.1666666665, 2017-04-13T23:59:59.999986, 235.4, 235.48, 232.29, 232.42, 1478385.0, 0, 234.51999999999998
          Data0, 0121, 736438.1666666665, 2017-04-18T23:59:59.999986, 232.5, 234.62, 232.4, 233.91, 1151612.0, 0, 234.17199999999997
          Data0, 0122, 736440.1666666665, 2017-04-20T23:59:59.999986, 234.05, 235.85, 233.18, 235.32, 1370622.0, 0, 234.37399999999997
          Data0, 0123, 736442.1666666665, 2017-04-22T23:59:59.999986, 235.45, 235.61, 234.0, 234.33, 733098.0, 0, 234.202
          Data0, 0124, 736446.1666666665, 2017-04-26T23:59:59.999986, 237.06, 239.53, 234.59, 238.44, 2268990.0, 0, 234.88400000000001
          Data0, 0125, 736448.1666666665, 2017-04-28T23:59:59.999986, 238.4, 238.96, 237.83, 237.93, 1057622.0, 0, 235.98600000000002
          1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators last edited by

            There was another thread which already addressed this problem, which is due to the rounding around midnight.

            The sample might be in need of refreshing, due to the stricter usage of sessionend when for example doing resampling operations. ibtest was created long before that was done to actually be safer in synchronizing data feeds.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors