next() not generated anymore after "IB and Trader Workstation connectivity lost"



  • Hello,

    I am connecting to IB for live data, everyday around 12:00pm HKT, the following error is received:

     ***** STORE NOTIF: <currentTime time=1490330404>
    <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>
    <tickPrice tickerId=16777217, field=1, price=111.332, canAutoExecute=1>
    

    After receiving the 1100 error message, the program continue to receive tickPrice and tickSize events but the notify_data() and notify_store() events never resumed and next() event is also never get generated. It is not resumed event after receiving the 1102 message from IB "Connectivity between IB and Trader Workstation has been restore - data maintained". To resume, I have to kill the program and restart from fresh.

    Has anyone got similar problem and if there is a way to resume the data feed in this scenario without restarting the program?

    Thanks very much for your help!


  • administrators

    See below for a disconnection/reconnection status executed a few minutes ago in which the codes 1100 and 1102 show up. The connection was kept down for about 1 minute.

    The puzzling thing from your message is:

    After receiving the 1100 error message, the program continue to receive tickPrice and tickSize events

    Which should not be possible because the connectivity between TWS and the world has been lost.

    This issue is probably related: https://github.com/mementum/backtrader/issues/283

    You mention a specific time which is 12:00 pm HKT and a quick guess, as also pointed out in the ticket, is that you are using something like IB Controller to keep TWS uninterruptedly. And/or that this time is the time at which the IB Server is reset every day. (HKT is GMT +8, so 12:00pm can easily match 06:00 which is the usual time for a reset in CET time)

    A 2nd quick guess would be that the internal tickers used for data identification with TWS are not surviving the resets/restarts mentioned above.

    $ ./ibtest.py --port 7497 --data0 EUR.JPY-CASH-IDEALPRO --resample --timeframe Seconds --compression 5 --no-backfill_start
    Server Version: 76
    TWS Time at connection:20170325 22:40:14 CET
    --------------------------------------------------
    Strategy Created
    --------------------------------------------------
    Timezone from ContractDetails: EST5EDT
    Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
    ***** DATA NOTIF: LIVE
    Data0, 0001, 736413.902951, 2017-03-25T17:40:15.000000, 122.635, 122.636, 122.635, 122.636, 0.0, 0, nan
    Data0, 0002, 736413.903009, 2017-03-25T17:40:20.000000, 122.637, 122.637, 122.637, 122.637, 0.0, 0, nan
    Data0, 0003, 736413.903067, 2017-03-25T17:40:25.000000, 122.609, 122.609, 122.609, 122.609, 0.0, 0, nan
    Data0, 0004, 736413.903125, 2017-03-25T17:40:30.000000, 122.61, 122.61, 122.61, 122.61, 0.0, 0, nan
    Data0, 0005, 736413.903183, 2017-03-25T17:40:35.000000, 122.608, 122.608, 122.608, 122.608, 0.0, 0, 122.62
    Data0, 0006, 736413.903241, 2017-03-25T17:40:40.000000, 122.592, 122.592, 122.592, 122.592, 0.0, 0, 122.6112
    ***** STORE NOTIF: <error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:ibdemo>
    ***** STORE NOTIF: <error id=-1, errorCode=2105, errorMsg=HMDS data farm connection is broken:demohmds>
    ***** STORE NOTIF: <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and TWS has been lost.>
    Data0, 0007, 736413.903299, 2017-03-25T17:40:45.000000, 122.561, 122.561, 122.561, 122.561, 0.0, 0, 122.596
    ***** STORE NOTIF: <error id=-1, errorCode=1102, errorMsg=Connectivity between IB and TWS has been restored - data maintained.>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
    ***** DATA NOTIF: DELAYED
    Data0, 0008, 736413.903356, 2017-03-25T17:40:50.000000, 119.78, 119.78, 119.78, 119.78, -1.0, 0, 122.0302
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
    Data0, 0009, 736413.903414, 2017-03-25T17:40:55.000000, 119.78, 119.785, 119.78, 119.78, -1.0, 0, 121.4642
    Data0, 0010, 736413.903472, 2017-03-25T17:41:00.000000, 119.78, 119.785, 119.775, 119.78, -1.0, 0, 120.8986
    Data0, 0011, 736413.90353, 2017-03-25T17:41:05.000000, 119.78, 119.78, 119.78, 119.78, -1.0, 0, 120.3362
    Data0, 0012, 736413.903588, 2017-03-25T17:41:10.000000, 119.78, 119.78, 119.755, 119.755, -1.0, 0, 119.775
    Data0, 0013, 736413.903646, 2017-03-25T17:41:15.000000, 119.755, 119.765, 119.755, 119.765, -1.0, 0, 119.772
    Data0, 0014, 736413.903704, 2017-03-25T17:41:20.000000, 119.765, 119.765, 119.76, 119.76, -1.0, 0, 119.768
    Data0, 0015, 736413.903762, 2017-03-25T17:41:25.000000, 119.76, 119.765, 119.76, 119.76, -1.0, 0, 119.764
    Data0, 0016, 736413.903819, 2017-03-25T17:41:30.000000, 119.76, 119.765, 119.76, 119.76, -1.0, 0, 119.76
    Data0, 0017, 736413.903877, 2017-03-25T17:41:35.000000, 119.76, 119.76, 119.76, 119.76, -1.0, 0, 119.761
    ***** DATA NOTIF: LIVE
    Data0, 0018, 736413.903935, 2017-03-25T17:41:40.000000, 122.616, 122.616, 122.615, 122.615, 0.0, 0, 120.331
    Data0, 0019, 736413.903993, 2017-03-25T17:41:45.000000, 122.616, 122.616, 122.616, 122.616, 0.0, 0, 120.9022
    Data0, 0020, 736413.904051, 2017-03-25T17:41:50.000000, 122.615, 122.644, 122.615, 122.644, 0.0, 0, 121.479
    Data0, 0021, 736413.904109, 2017-03-25T17:41:55.000000, 122.646, 122.647, 122.646, 122.647, 0.0, 0, 122.0564
    Data0, 0022, 736413.904167, 2017-03-25T17:42:00.000000, 122.614, 122.614, 122.614, 122.614, 0.0, 0, 122.6272
    Data0, 0023, 736413.904225, 2017-03-25T17:42:05.000000, 122.615, 122.615, 122.615, 122.615, 0.0, 0, 122.6272
    Data0, 0024, 736413.904282, 2017-03-25T17:42:10.000000, 122.628, 122.629, 122.628, 122.629, 0.0, 0, 122.6298
    


  • I also believe the daily disconnection was due to IB server reset. Thanks for the pointer to issue #283, I will further check on that.

    Below is the log related to the puzzle: IB and TWS disconnected but program continue to receive tickPrice and tickSize events. There is no more STORE NOTIF logged after the errorCode=1100 TWS to IB disconnection or even after received errorCode=1102 TWS to IB connection resumed. It probably also means that cerebro and strategy is not receiving any of this ticks after 1100 error there no next() event was generated....

    ***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=1000000>
    ***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=1000000>
    <currentTime time=1490330404>
    ***** STORE NOTIF: <currentTime time=1490330404>
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    <tickPrice tickerId=16777217, field=1, price=111.332, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickPrice tickerId=16777217, field=2, price=111.334, canAutoExecute=1>
    <tickSize tickerId=16777217, field=3, size=3000000>
    <tickSize tickerId=16777217, field=3, size=3000000>
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickPrice tickerId=16777217, field=1, price=111.333, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickPrice tickerId=16777217, field=2, price=111.335, canAutoExecute=1>
    <tickSize tickerId=16777217, field=3, size=7000000>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickSize tickerId=16777217, field=3, size=7000000>
    <tickSize tickerId=16777217, field=3, size=5000000>
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    <tickSize tickerId=16777217, field=3, size=3000000>
    <tickSize tickerId=16777217, field=0, size=2000000>
    <tickSize tickerId=16777217, field=0, size=3000000>
    <tickSize tickerId=16777217, field=3, size=2000000>
    <tickSize tickerId=16777217, field=3, size=3000000>
    <tickPrice tickerId=16777217, field=1, price=111.334, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickSize tickerId=16777217, field=3, size=2000000>
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickPrice tickerId=16777217, field=2, price=111.336, canAutoExecute=1>
    ...
    ...
    <error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:cashfarm>
    **<error id=-1, errorCode=1102, errorMsg=Connectivity between IB and Trader Workstation has been restored - data maintained.>**
    <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
    <tickPrice tickerId=16777217, field=6, price=111.445, canAutoExecute=0>
    <tickPrice tickerId=16777217, field=7, price=110.865, canAutoExecute=0>
    <tickPrice tickerId=16777217, field=9, price=110.92, canAutoExecute=0>
    <tickPrice tickerId=16777217, field=1, price=111.328, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=4000000>
    <tickPrice tickerId=16777217, field=2, price=111.33, canAutoExecute=1>
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickSize tickerId=16777217, field=0, size=4000000>
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickPrice tickerId=16777217, field=1, price=111.329, canAutoExecute=1>
    


  • The following links could help solve the problem:
    https://github.com/erdewit/tws_async

    tws_async

    Integrate the Python IB API from Interactive Brokers with asyncio.

    Unix

    Unix users can use the tws_async.py script. It requires Python version >= 3.5 and the Interactive Brokers API.

    Windows

    Windows users are encouraged to use the tws_async_windows.py script that bypasses some reconnect issues.


  • administrators

    No. There is no reconnect issue, because the connection has not been lost. It simply seems that the reset of the TWS servers affects the tickers. And that's for the IB Python API which is not a target in this project for the Python >= 3.x constraint.

    It will be considered in future versions.



  • @backtrader I tried restarting bt after running into this issue and that seems to make it work, so am using this as a workaround.

    However I think we should add some logic into backtrader to automatically detect and recover from this, as this effectively removes the ability to run any algo for more than a day against IB.


  • administrators

    Ideally yes.


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.