For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

next() not generated anymore after "IB and Trader Workstation connectivity lost"



  • Hello,

    I am connecting to IB for live data, everyday around 12:00pm HKT, the following error is received:

     ***** STORE NOTIF: <currentTime time=1490330404>
    <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>
    <tickPrice tickerId=16777217, field=1, price=111.332, canAutoExecute=1>
    

    After receiving the 1100 error message, the program continue to receive tickPrice and tickSize events but the notify_data() and notify_store() events never resumed and next() event is also never get generated. It is not resumed event after receiving the 1102 message from IB "Connectivity between IB and Trader Workstation has been restore - data maintained". To resume, I have to kill the program and restart from fresh.

    Has anyone got similar problem and if there is a way to resume the data feed in this scenario without restarting the program?

    Thanks very much for your help!


  • administrators

    See below for a disconnection/reconnection status executed a few minutes ago in which the codes 1100 and 1102 show up. The connection was kept down for about 1 minute.

    The puzzling thing from your message is:

    After receiving the 1100 error message, the program continue to receive tickPrice and tickSize events

    Which should not be possible because the connectivity between TWS and the world has been lost.

    This issue is probably related: https://github.com/mementum/backtrader/issues/283

    You mention a specific time which is 12:00 pm HKT and a quick guess, as also pointed out in the ticket, is that you are using something like IB Controller to keep TWS uninterruptedly. And/or that this time is the time at which the IB Server is reset every day. (HKT is GMT +8, so 12:00pm can easily match 06:00 which is the usual time for a reset in CET time)

    A 2nd quick guess would be that the internal tickers used for data identification with TWS are not surviving the resets/restarts mentioned above.

    $ ./ibtest.py --port 7497 --data0 EUR.JPY-CASH-IDEALPRO --resample --timeframe Seconds --compression 5 --no-backfill_start
    Server Version: 76
    TWS Time at connection:20170325 22:40:14 CET
    --------------------------------------------------
    Strategy Created
    --------------------------------------------------
    Timezone from ContractDetails: EST5EDT
    Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
    ***** DATA NOTIF: LIVE
    Data0, 0001, 736413.902951, 2017-03-25T17:40:15.000000, 122.635, 122.636, 122.635, 122.636, 0.0, 0, nan
    Data0, 0002, 736413.903009, 2017-03-25T17:40:20.000000, 122.637, 122.637, 122.637, 122.637, 0.0, 0, nan
    Data0, 0003, 736413.903067, 2017-03-25T17:40:25.000000, 122.609, 122.609, 122.609, 122.609, 0.0, 0, nan
    Data0, 0004, 736413.903125, 2017-03-25T17:40:30.000000, 122.61, 122.61, 122.61, 122.61, 0.0, 0, nan
    Data0, 0005, 736413.903183, 2017-03-25T17:40:35.000000, 122.608, 122.608, 122.608, 122.608, 0.0, 0, 122.62
    Data0, 0006, 736413.903241, 2017-03-25T17:40:40.000000, 122.592, 122.592, 122.592, 122.592, 0.0, 0, 122.6112
    ***** STORE NOTIF: <error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:ibdemo>
    ***** STORE NOTIF: <error id=-1, errorCode=2105, errorMsg=HMDS data farm connection is broken:demohmds>
    ***** STORE NOTIF: <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and TWS has been lost.>
    Data0, 0007, 736413.903299, 2017-03-25T17:40:45.000000, 122.561, 122.561, 122.561, 122.561, 0.0, 0, 122.596
    ***** STORE NOTIF: <error id=-1, errorCode=1102, errorMsg=Connectivity between IB and TWS has been restored - data maintained.>
    ***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
    ***** DATA NOTIF: DELAYED
    Data0, 0008, 736413.903356, 2017-03-25T17:40:50.000000, 119.78, 119.78, 119.78, 119.78, -1.0, 0, 122.0302
    ***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
    Data0, 0009, 736413.903414, 2017-03-25T17:40:55.000000, 119.78, 119.785, 119.78, 119.78, -1.0, 0, 121.4642
    Data0, 0010, 736413.903472, 2017-03-25T17:41:00.000000, 119.78, 119.785, 119.775, 119.78, -1.0, 0, 120.8986
    Data0, 0011, 736413.90353, 2017-03-25T17:41:05.000000, 119.78, 119.78, 119.78, 119.78, -1.0, 0, 120.3362
    Data0, 0012, 736413.903588, 2017-03-25T17:41:10.000000, 119.78, 119.78, 119.755, 119.755, -1.0, 0, 119.775
    Data0, 0013, 736413.903646, 2017-03-25T17:41:15.000000, 119.755, 119.765, 119.755, 119.765, -1.0, 0, 119.772
    Data0, 0014, 736413.903704, 2017-03-25T17:41:20.000000, 119.765, 119.765, 119.76, 119.76, -1.0, 0, 119.768
    Data0, 0015, 736413.903762, 2017-03-25T17:41:25.000000, 119.76, 119.765, 119.76, 119.76, -1.0, 0, 119.764
    Data0, 0016, 736413.903819, 2017-03-25T17:41:30.000000, 119.76, 119.765, 119.76, 119.76, -1.0, 0, 119.76
    Data0, 0017, 736413.903877, 2017-03-25T17:41:35.000000, 119.76, 119.76, 119.76, 119.76, -1.0, 0, 119.761
    ***** DATA NOTIF: LIVE
    Data0, 0018, 736413.903935, 2017-03-25T17:41:40.000000, 122.616, 122.616, 122.615, 122.615, 0.0, 0, 120.331
    Data0, 0019, 736413.903993, 2017-03-25T17:41:45.000000, 122.616, 122.616, 122.616, 122.616, 0.0, 0, 120.9022
    Data0, 0020, 736413.904051, 2017-03-25T17:41:50.000000, 122.615, 122.644, 122.615, 122.644, 0.0, 0, 121.479
    Data0, 0021, 736413.904109, 2017-03-25T17:41:55.000000, 122.646, 122.647, 122.646, 122.647, 0.0, 0, 122.0564
    Data0, 0022, 736413.904167, 2017-03-25T17:42:00.000000, 122.614, 122.614, 122.614, 122.614, 0.0, 0, 122.6272
    Data0, 0023, 736413.904225, 2017-03-25T17:42:05.000000, 122.615, 122.615, 122.615, 122.615, 0.0, 0, 122.6272
    Data0, 0024, 736413.904282, 2017-03-25T17:42:10.000000, 122.628, 122.629, 122.628, 122.629, 0.0, 0, 122.6298
    


  • I also believe the daily disconnection was due to IB server reset. Thanks for the pointer to issue #283, I will further check on that.

    Below is the log related to the puzzle: IB and TWS disconnected but program continue to receive tickPrice and tickSize events. There is no more STORE NOTIF logged after the errorCode=1100 TWS to IB disconnection or even after received errorCode=1102 TWS to IB connection resumed. It probably also means that cerebro and strategy is not receiving any of this ticks after 1100 error there no next() event was generated....

    ***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=1000000>
    ***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=1000000>
    <currentTime time=1490330404>
    ***** STORE NOTIF: <currentTime time=1490330404>
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    <tickPrice tickerId=16777217, field=1, price=111.332, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickPrice tickerId=16777217, field=2, price=111.334, canAutoExecute=1>
    <tickSize tickerId=16777217, field=3, size=3000000>
    <tickSize tickerId=16777217, field=3, size=3000000>
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickPrice tickerId=16777217, field=1, price=111.333, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickSize tickerId=16777217, field=0, size=500000>
    <tickPrice tickerId=16777217, field=2, price=111.335, canAutoExecute=1>
    <tickSize tickerId=16777217, field=3, size=7000000>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickSize tickerId=16777217, field=3, size=7000000>
    <tickSize tickerId=16777217, field=3, size=5000000>
    **<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
    <tickSize tickerId=16777217, field=3, size=3000000>
    <tickSize tickerId=16777217, field=0, size=2000000>
    <tickSize tickerId=16777217, field=0, size=3000000>
    <tickSize tickerId=16777217, field=3, size=2000000>
    <tickSize tickerId=16777217, field=3, size=3000000>
    <tickPrice tickerId=16777217, field=1, price=111.334, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickSize tickerId=16777217, field=0, size=1000000>
    <tickSize tickerId=16777217, field=3, size=2000000>
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickPrice tickerId=16777217, field=2, price=111.336, canAutoExecute=1>
    ...
    ...
    <error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:cashfarm>
    **<error id=-1, errorCode=1102, errorMsg=Connectivity between IB and Trader Workstation has been restored - data maintained.>**
    <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
    <tickPrice tickerId=16777217, field=6, price=111.445, canAutoExecute=0>
    <tickPrice tickerId=16777217, field=7, price=110.865, canAutoExecute=0>
    <tickPrice tickerId=16777217, field=9, price=110.92, canAutoExecute=0>
    <tickPrice tickerId=16777217, field=1, price=111.328, canAutoExecute=1>
    <tickSize tickerId=16777217, field=0, size=4000000>
    <tickPrice tickerId=16777217, field=2, price=111.33, canAutoExecute=1>
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickSize tickerId=16777217, field=0, size=4000000>
    <tickSize tickerId=16777217, field=3, size=1000000>
    <tickPrice tickerId=16777217, field=1, price=111.329, canAutoExecute=1>
    


  • The following links could help solve the problem:
    https://github.com/erdewit/tws_async

    tws_async

    Integrate the Python IB API from Interactive Brokers with asyncio.

    Unix

    Unix users can use the tws_async.py script. It requires Python version >= 3.5 and the Interactive Brokers API.

    Windows

    Windows users are encouraged to use the tws_async_windows.py script that bypasses some reconnect issues.


  • administrators

    No. There is no reconnect issue, because the connection has not been lost. It simply seems that the reset of the TWS servers affects the tickers. And that's for the IB Python API which is not a target in this project for the Python >= 3.x constraint.

    It will be considered in future versions.



  • @backtrader I tried restarting bt after running into this issue and that seems to make it work, so am using this as a workaround.

    However I think we should add some logic into backtrader to automatically detect and recover from this, as this effectively removes the ability to run any algo for more than a day against IB.


  • administrators

    Ideally yes.



  • @jaikumarm Did you ever find a solution for this? I'm running across the same problem.



  • @backtrader @jaikumarm I'm trying to solve this problem because restarting backtrader is not an option for me (live trades, etc). Could you guide me on where to look to solve the issue?



  • Is there any solution for this issue?

    @you @Guest have you managed to figure it out somehow?


  • administrators

    Not really. The reproduction attempts failed.



  • @backtrader what did you do to reproduce this issue?

    I will try to reproduce and provide the results/logs you could need. What will you need to analyze the problem besides API logs/client logs?


  • administrators

    Wait for "IB and Trader Workstation connectivity lost"`?



  • I print out the prices every 30 secs and every morning when I check the printout, it stops at some time ( today it was 04:38 utc ). There is no warning/error, the client is still connected to the IB Gateway. In parallel I am running another client app with ib_insync, it keeps streaming.



  • @backtrader I have experienced the same thing again today at exactly same time (04:38:30). I believe this is due to IB server restart or similar. backtrader stopped streaming again. I had the same problem with my other client in parallel which uses ib_insync. ib_insync client gave an AttributeError saying marketPrice of contract was not available. So I just resubscribed with reqMktData. The stream continued with ib_insync. Does backtrader resubscribe?



  • and here is the log. backtrader stops streaming prices after that.

    AUD.CAD-CASH-IDEALPRO 2018-08-17 04:37:30 0.95555
    AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:37:30 0.95555
    AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:37:30 0.95555
    
    
    Server Version: 76
    TWS Time at connection:20180817 04:38:01 Eastern European Time
    ***** 2018-08-17 04:38:05.929322 DATA NOTIF: CONNBROKEN
    AUD.CAD-CASH-IDEALPRO
    ***** 2018-08-17 04:38:05.929374 DATA NOTIF: DELAYED
    AUD.CAD-CASH-IDEALPRO
    AUD.CAD-CASH-IDEALPRO 2018-08-17 04:38:00 0.95559
    AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:38:00 0.95559
    AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:38:00 0.95559
    
    
    ***** 2018-08-17 04:38:05.935522 STORE NOTIF: <error id=None, errorCode=None, errorMsg=unpack requires a buffer of 1 bytes>
    ***** 2018-08-17 04:38:05.935564 STORE NOTIF: <error id=-1, errorCode=504, errorMsg=Not connected>
    
    ***** 2018-08-17 04:38:05.936275 STORE NOTIF: <error id=-1, errorCode=502, errorMsg=Couldn't connect to TWS.  Confirm that "Enable ActiveX and Socket Clients" is enabled on the TWS "Configure->API" menu.>
    ***** 2018-08-17 04:38:05.936309 STORE NOTIF: <error id=-1, errorCode=502, errorMsg=Couldn't connect to TWS.  Confirm that "Enable ActiveX and Socket Clients" is enabled on the TWS "Configure->API" menu.>
    ***** 2018-08-17 04:38:05.936346 STORE NOTIF: <error id=-1, errorCode=502, errorMsg=Couldn't connect to TWS.  Confirm that "Enable ActiveX and Socket Clients" is enabled on the TWS "Configure->API" menu.>
    ***** 2018-08-17 04:38:05.936384 STORE NOTIF: <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:cashfarm>
    ***** 2018-08-17 04:38:05.936408 STORE NOTIF: <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:cashfarm>
    ***** 2018-08-17 04:38:05.936437 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
    ***** 2018-08-17 04:38:05.936471 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
    ***** 2018-08-17 04:38:05.941009 DATA NOTIF: LIVE
    AUD.CAD-CASH-IDEALPRO
    AUD.CAD-CASH-IDEALPRO 2018-08-17 04:38:30 0.9556
    AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:38:30 0.9556
    AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:38:30 0.9556