next() not generated anymore after "IB and Trader Workstation connectivity lost"

you

Hello,

I am connecting to IB for live data, everyday around 12:00pm HKT, the following error is received:

 ***** STORE NOTIF: <currentTime time=1490330404>
<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>
<tickPrice tickerId=16777217, field=1, price=111.332, canAutoExecute=1>

After receiving the 1100 error message, the program continue to receive tickPrice and tickSize events but the notify_data() and notify_store() events never resumed and next() event is also never get generated. It is not resumed event after receiving the 1102 message from IB "Connectivity between IB and Trader Workstation has been restore - data maintained". To resume, I have to kill the program and restart from fresh.

Has anyone got similar problem and if there is a way to resume the data feed in this scenario without restarting the program?

Thanks very much for your help!

backtrader

See below for a disconnection/reconnection status executed a few minutes ago in which the codes 1100 and 1102 show up. The connection was kept down for about 1 minute.

The puzzling thing from your message is:

After receiving the 1100 error message, the program continue to receive tickPrice and tickSize events

Which should not be possible because the connectivity between TWS and the world has been lost.

This issue is probably related: https://github.com/mementum/backtrader/issues/283

You mention a specific time which is 12:00 pm HKT and a quick guess, as also pointed out in the ticket, is that you are using something like IB Controller to keep TWS uninterruptedly. And/or that this time is the time at which the IB Server is reset every day. (HKT is GMT +8, so 12:00pm can easily match 06:00 which is the usual time for a reset in CET time)

A 2nd quick guess would be that the internal tickers used for data identification with TWS are not surviving the resets/restarts mentioned above.

$ ./ibtest.py --port 7497 --data0 EUR.JPY-CASH-IDEALPRO --resample --timeframe Seconds --compression 5 --no-backfill_start
Server Version: 76
TWS Time at connection:20170325 22:40:14 CET
--------------------------------------------------
Strategy Created
--------------------------------------------------
Timezone from ContractDetails: EST5EDT
Datetime, Open, High, Low, Close, Volume, OpenInterest, SMA
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
***** DATA NOTIF: LIVE
Data0, 0001, 736413.902951, 2017-03-25T17:40:15.000000, 122.635, 122.636, 122.635, 122.636, 0.0, 0, nan
Data0, 0002, 736413.903009, 2017-03-25T17:40:20.000000, 122.637, 122.637, 122.637, 122.637, 0.0, 0, nan
Data0, 0003, 736413.903067, 2017-03-25T17:40:25.000000, 122.609, 122.609, 122.609, 122.609, 0.0, 0, nan
Data0, 0004, 736413.903125, 2017-03-25T17:40:30.000000, 122.61, 122.61, 122.61, 122.61, 0.0, 0, nan
Data0, 0005, 736413.903183, 2017-03-25T17:40:35.000000, 122.608, 122.608, 122.608, 122.608, 0.0, 0, 122.62
Data0, 0006, 736413.903241, 2017-03-25T17:40:40.000000, 122.592, 122.592, 122.592, 122.592, 0.0, 0, 122.6112
***** STORE NOTIF: <error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:ibdemo>
***** STORE NOTIF: <error id=-1, errorCode=2105, errorMsg=HMDS data farm connection is broken:demohmds>
***** STORE NOTIF: <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and TWS has been lost.>
Data0, 0007, 736413.903299, 2017-03-25T17:40:45.000000, 122.561, 122.561, 122.561, 122.561, 0.0, 0, 122.596
***** STORE NOTIF: <error id=-1, errorCode=1102, errorMsg=Connectivity between IB and TWS has been restored - data maintained.>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
***** DATA NOTIF: DELAYED
Data0, 0008, 736413.903356, 2017-03-25T17:40:50.000000, 119.78, 119.78, 119.78, 119.78, -1.0, 0, 122.0302
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
Data0, 0009, 736413.903414, 2017-03-25T17:40:55.000000, 119.78, 119.785, 119.78, 119.78, -1.0, 0, 121.4642
Data0, 0010, 736413.903472, 2017-03-25T17:41:00.000000, 119.78, 119.785, 119.775, 119.78, -1.0, 0, 120.8986
Data0, 0011, 736413.90353, 2017-03-25T17:41:05.000000, 119.78, 119.78, 119.78, 119.78, -1.0, 0, 120.3362
Data0, 0012, 736413.903588, 2017-03-25T17:41:10.000000, 119.78, 119.78, 119.755, 119.755, -1.0, 0, 119.775
Data0, 0013, 736413.903646, 2017-03-25T17:41:15.000000, 119.755, 119.765, 119.755, 119.765, -1.0, 0, 119.772
Data0, 0014, 736413.903704, 2017-03-25T17:41:20.000000, 119.765, 119.765, 119.76, 119.76, -1.0, 0, 119.768
Data0, 0015, 736413.903762, 2017-03-25T17:41:25.000000, 119.76, 119.765, 119.76, 119.76, -1.0, 0, 119.764
Data0, 0016, 736413.903819, 2017-03-25T17:41:30.000000, 119.76, 119.765, 119.76, 119.76, -1.0, 0, 119.76
Data0, 0017, 736413.903877, 2017-03-25T17:41:35.000000, 119.76, 119.76, 119.76, 119.76, -1.0, 0, 119.761
***** DATA NOTIF: LIVE
Data0, 0018, 736413.903935, 2017-03-25T17:41:40.000000, 122.616, 122.616, 122.615, 122.615, 0.0, 0, 120.331
Data0, 0019, 736413.903993, 2017-03-25T17:41:45.000000, 122.616, 122.616, 122.616, 122.616, 0.0, 0, 120.9022
Data0, 0020, 736413.904051, 2017-03-25T17:41:50.000000, 122.615, 122.644, 122.615, 122.644, 0.0, 0, 121.479
Data0, 0021, 736413.904109, 2017-03-25T17:41:55.000000, 122.646, 122.647, 122.646, 122.647, 0.0, 0, 122.0564
Data0, 0022, 736413.904167, 2017-03-25T17:42:00.000000, 122.614, 122.614, 122.614, 122.614, 0.0, 0, 122.6272
Data0, 0023, 736413.904225, 2017-03-25T17:42:05.000000, 122.615, 122.615, 122.615, 122.615, 0.0, 0, 122.6272
Data0, 0024, 736413.904282, 2017-03-25T17:42:10.000000, 122.628, 122.629, 122.628, 122.629, 0.0, 0, 122.6298

you

I also believe the daily disconnection was due to IB server reset. Thanks for the pointer to issue #283, I will further check on that.

Below is the log related to the puzzle: IB and TWS disconnected but program continue to receive tickPrice and tickSize events. There is no more STORE NOTIF logged after the errorCode=1100 TWS to IB disconnection or even after received errorCode=1102 TWS to IB connection resumed. It probably also means that cerebro and strategy is not receiving any of this ticks after 1100 error there no next() event was generated....

***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=1000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=1000000>
<currentTime time=1490330404>
***** STORE NOTIF: <currentTime time=1490330404>
**<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
<tickPrice tickerId=16777217, field=1, price=111.332, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=500000>
<tickSize tickerId=16777217, field=0, size=500000>
<tickSize tickerId=16777217, field=0, size=1000000>
<tickPrice tickerId=16777217, field=2, price=111.334, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=3000000>
<tickSize tickerId=16777217, field=3, size=3000000>
**<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
**<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
<tickSize tickerId=16777217, field=3, size=1000000>
<tickPrice tickerId=16777217, field=1, price=111.333, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=500000>
<tickSize tickerId=16777217, field=0, size=500000>
<tickPrice tickerId=16777217, field=2, price=111.335, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=7000000>
<tickSize tickerId=16777217, field=0, size=1000000>
<tickSize tickerId=16777217, field=3, size=7000000>
<tickSize tickerId=16777217, field=3, size=5000000>
**<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>**
<tickSize tickerId=16777217, field=3, size=3000000>
<tickSize tickerId=16777217, field=0, size=2000000>
<tickSize tickerId=16777217, field=0, size=3000000>
<tickSize tickerId=16777217, field=3, size=2000000>
<tickSize tickerId=16777217, field=3, size=3000000>
<tickPrice tickerId=16777217, field=1, price=111.334, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=1000000>
<tickSize tickerId=16777217, field=0, size=1000000>
<tickSize tickerId=16777217, field=3, size=2000000>
<tickSize tickerId=16777217, field=3, size=1000000>
<tickPrice tickerId=16777217, field=2, price=111.336, canAutoExecute=1>
...
...
<error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:cashfarm>
**<error id=-1, errorCode=1102, errorMsg=Connectivity between IB and Trader Workstation has been restored - data maintained.>**
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
<tickPrice tickerId=16777217, field=6, price=111.445, canAutoExecute=0>
<tickPrice tickerId=16777217, field=7, price=110.865, canAutoExecute=0>
<tickPrice tickerId=16777217, field=9, price=110.92, canAutoExecute=0>
<tickPrice tickerId=16777217, field=1, price=111.328, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=4000000>
<tickPrice tickerId=16777217, field=2, price=111.33, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=1000000>
<tickSize tickerId=16777217, field=0, size=4000000>
<tickSize tickerId=16777217, field=3, size=1000000>
<tickPrice tickerId=16777217, field=1, price=111.329, canAutoExecute=1>

zfsamzfsam

The following links could help solve the problem:
https://github.com/erdewit/tws_async

tws_async

Integrate the Python IB API from Interactive Brokers with asyncio.

Unix

Unix users can use the tws_async.py script. It requires Python version >= 3.5 and the Interactive Brokers API.

Windows

Windows users are encouraged to use the tws_async_windows.py script that bypasses some reconnect issues.

backtrader

No. There is no reconnect issue, because the connection has not been lost. It simply seems that the reset of the TWS servers affects the tickers. And that's for the IB Python API which is not a target in this project for the Python >= 3.x constraint.

It will be considered in future versions.

jaikumarm

@backtrader I tried restarting bt after running into this issue and that seems to make it work, so am using this as a workaround.

However I think we should add some logic into backtrader to automatically detect and recover from this, as this effectively removes the ability to run any algo for more than a day against IB.

backtrader

Ideally yes.

@jaikumarm Did you ever find a solution for this? I'm running across the same problem.

@backtrader @jaikumarm I'm trying to solve this problem because restarting backtrader is not an option for me (live trades, etc). Could you guide me on where to look to solve the issue?

bb2

Is there any solution for this issue?

@you @Guest have you managed to figure it out somehow?

backtrader

Not really. The reproduction attempts failed.

bb2

@backtrader what did you do to reproduce this issue?

I will try to reproduce and provide the results/logs you could need. What will you need to analyze the problem besides API logs/client logs?

backtrader

Wait for "IB and Trader Workstation connectivity lost"`?

bb2

I print out the prices every 30 secs and every morning when I check the printout, it stops at some time ( today it was 04:38 utc ). There is no warning/error, the client is still connected to the IB Gateway. In parallel I am running another client app with ib_insync, it keeps streaming.

bb2

@backtrader I have experienced the same thing again today at exactly same time (04:38:30). I believe this is due to IB server restart or similar. backtrader stopped streaming again. I had the same problem with my other client in parallel which uses ib_insync. ib_insync client gave an AttributeError saying marketPrice of contract was not available. So I just resubscribed with reqMktData. The stream continued with ib_insync. Does backtrader resubscribe?

bb2

and here is the log. backtrader stops streaming prices after that.

AUD.CAD-CASH-IDEALPRO 2018-08-17 04:37:30 0.95555
AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:37:30 0.95555
AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:37:30 0.95555


Server Version: 76
TWS Time at connection:20180817 04:38:01 Eastern European Time
***** 2018-08-17 04:38:05.929322 DATA NOTIF: CONNBROKEN
AUD.CAD-CASH-IDEALPRO
***** 2018-08-17 04:38:05.929374 DATA NOTIF: DELAYED
AUD.CAD-CASH-IDEALPRO
AUD.CAD-CASH-IDEALPRO 2018-08-17 04:38:00 0.95559
AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:38:00 0.95559
AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:38:00 0.95559


***** 2018-08-17 04:38:05.935522 STORE NOTIF: <error id=None, errorCode=None, errorMsg=unpack requires a buffer of 1 bytes>
***** 2018-08-17 04:38:05.935564 STORE NOTIF: <error id=-1, errorCode=504, errorMsg=Not connected>

***** 2018-08-17 04:38:05.936275 STORE NOTIF: <error id=-1, errorCode=502, errorMsg=Couldn't connect to TWS.  Confirm that "Enable ActiveX and Socket Clients" is enabled on the TWS "Configure->API" menu.>
***** 2018-08-17 04:38:05.936309 STORE NOTIF: <error id=-1, errorCode=502, errorMsg=Couldn't connect to TWS.  Confirm that "Enable ActiveX and Socket Clients" is enabled on the TWS "Configure->API" menu.>
***** 2018-08-17 04:38:05.936346 STORE NOTIF: <error id=-1, errorCode=502, errorMsg=Couldn't connect to TWS.  Confirm that "Enable ActiveX and Socket Clients" is enabled on the TWS "Configure->API" menu.>
***** 2018-08-17 04:38:05.936384 STORE NOTIF: <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:cashfarm>
***** 2018-08-17 04:38:05.936408 STORE NOTIF: <error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:cashfarm>
***** 2018-08-17 04:38:05.936437 STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
***** 2018-08-17 04:38:05.936471 STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:cashhmds>
***** 2018-08-17 04:38:05.941009 DATA NOTIF: LIVE
AUD.CAD-CASH-IDEALPRO
AUD.CAD-CASH-IDEALPRO 2018-08-17 04:38:30 0.9556
AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:38:30 0.9556
AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:38:30 0.9556

bb2

@backtrader could you share sample code which doesn't have any connection issues with IB described above?

backtrader

The only code used is in ibtest. Your log is unclear:

There is data at the beginning.

AUD.CAD-CASH-IDEALPRO 2018-08-17 04:37:30 0.95555
AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:37:30 0.95555
AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:37:30 0.95555

There is a Connection to the Server !!!!???
The data connection is then broken

Where does the data before the connection come from?
There is some delayed data which is obviously what can be saved by the resampler

@bb2 said in next() not generated anymore after "IB and Trader Workstation connectivity lost":

AUD.CAD-CASH-IDEALPRO
AUD.CAD-CASH-IDEALPRO 2018-08-17 04:38:00 0.95559
AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:38:00 0.95559
AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:38:00 0.95559

There is data after the disconnection (with the new LIVE)

@bb2 said in next() not generated anymore after "IB and Trader Workstation connectivity lost":

    ***** 2018-08-17 04:38:05.941009 DATA NOTIF: LIVE
    AUD.CAD-CASH-IDEALPRO
    AUD.CAD-CASH-IDEALPRO 2018-08-17 04:38:30 0.9556
    AUD.CAD-CASH-IDEALPRO_1m 2018-08-17 04:38:30 0.9556
AUD.CAD-CASH-IDEALPRO_15m 2018-08-17 04:38:30 0.9556

And the timestamps keep on growing.

vytautas a

Hello,
I am encountering a similar issue with backtrader failing to resume firing next() after the IB server restart. Here is the scenario.
Before the IB server restart everything goes normal - store notifications arrive as well as tickSize and tickPrice events:

***** STORE NOTIF: <tickSize tickerId=16777217, field=0, size=1000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=8000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=10000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=13000000>
<tickSize tickerId=16777217, field=0, size=3000000>
<tickSize tickerId=16777217, field=3, size=11000000>
<tickSize tickerId=16777217, field=0, size=6000000>
<tickSize tickerId=16777217, field=3, size=8000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=0, size=3000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=11000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=0, size=6000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=8000000>
<tickSize tickerId=16777217, field=3, size=9000000>
***** STORE NOTIF: <tickSize tickerId=16777217, field=3, size=9000000>
<tickSize tickerId=16777217, field=0, size=4000000>
<tickPrice tickerId=16777217, field=1, price=1.13914, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=2000000>
<tickSize tickerId=16777217, field=0, size=2000000>
<tickSize tickerId=16777217, field=3, size=4500000>
<tickSize tickerId=16777217, field=3, size=3000000>
<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>
<tickSize tickerId=16777217, field=3, size=4500000>
<tickPrice tickerId=16777217, field=1, price=1.13913, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=4000000>
<tickPrice tickerId=16777217, field=2, price=1.13914, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=3000000>
<tickSize tickerId=16777217, field=0, size=4000000>
<tickSize tickerId=16777217, field=3, size=3000000>
<error id=-1, errorCode=1100, errorMsg=Connectivity between IB and Trader Workstation has been lost.>
<tickPrice tickerId=16777217, field=1, price=1.13914, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=2000000>
<tickPrice tickerId=16777217, field=2, price=1.13915, canAutoExecute=1>
<tickSize tickerId=16777217, field=3, size=4500000>

However, as you can see from the log snippet above, after the errorCode=1100 no more STORE NOTIF is recorded and store notifications do not appear even after the connection is restored:

<tickSize tickerId=16777217, field=0, size=5000000>
<tickSize tickerId=16777217, field=3, size=500000>
<tickPrice tickerId=16777217, field=1, price=1.13905, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=2000000>
<tickSize tickerId=16777217, field=0, size=2000000>
<tickSize tickerId=16777217, field=3, size=1500000>
<tickSize tickerId=16777217, field=3, size=2000000>
<tickSize tickerId=16777217, field=3, size=4000000>
<tickPrice tickerId=16777217, field=1, price=1.13904, canAutoExecute=1>
<tickSize tickerId=16777217, field=0, size=2000000>
<error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:njcash>
<error id=-1, errorCode=1102, errorMsg=Connectivity between IB and Trader Workstation has been restored - data maintained.>
<updateAccountTime timeStamp=08:03>
<updateAccountTime timeStamp=08:03>
<updateAccountValue key=AccruedCash, value=846.90, currency=USD, accountName=******>
<updateAccountTime timeStamp=08:03>
<updateAccountTime timeStamp=08:03>
<updateAccountTime timeStamp=08:03>
<updateAccountValue key=AccruedCash, value=847, currency=USD, accountName=******>
<updateAccountTime timeStamp=08:03>
<accountDownloadEnd accountName=******>
<error id=-1, errorCode=2119, errorMsg=Market data farm is connecting:cashfarm>
<error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:cashfarm>
<tickPrice tickerId=16777217, field=6, price=1.14005, canAutoExecute=0>
<tickPrice tickerId=16777217, field=7, price=1.13845, canAutoExecute=0>

As you can see the tick events have never stopped appearing. So basically everything is the same as @you and @bb2 described.
Moreover, I have tried simply disabling the internet connection while the backtrader is running and in this case after reconnection everything goes back to normal after a few minutes. So it seems that the problem is associated only with the IB server restart and not the disappearing internet connection.

Thank you for your help.

vytautas a

It seems that I have found a way to solve the above-mentioned issue. I have dug into the code in ibdata.py file which constructs data feeds for backtrader using IB API. What I found is that the whole problem is contained in the _load function. During normal operation the following code gets messages msg from the live queue and further code in the infinite loop of _load processes this message.

try:
    msg = (self._storedmsg.pop(None, None) or
           self.qlive.get(timeout=self._qcheck))   
except queue.Empty:
    if True:
        return None

During the IB server restart I noticed that for some reason the message containing error code -1100 (connection lost) is followed by another message containing RTVolume object which would indicate that connection is still operational and data still arrives. However, as previously a message with error -1100 arrived, the variable self._statelivereconn was assigned value True which would indicate that historical backfilling is needed. Now, as in reality connection is still down the request for historical data could not possibly be successful. Thus, the hole code hangs when waiting for something to come out of self.qhist queue in the following piece of code:

elif self._state == self._ST_HISTORBACK:
    msg = self.qhist.get()
    ...

.get() method never receives anything from the historical queue and waits forever. Why doesn't the queue get something when connection to IB servers is reestablished is beyond me.
Now for the solution. It might not be the best or most elegant but it worked for me. What I did is to explicitly prohibit processing any message after message with error -1100 arrived:

while True:
    if self._state == self._ST_LIVE:
        try:
            msg = (self._storedmsg.pop(None, None) or
                           self.qlive.get(timeout=self._qcheck))
        except queue.Empty:
            if True:
                return None

            # Code invalidated until further checking is done
            if not self._statelivereconn:
                return None  # indicate timeout situation

            # Awaiting data and nothing came in - fake it up until now
            dtend = self.num2date(date2num(datetime.datetime.utcnow()))
            dtbegin = None
            if len(self) > 1:
                dtbegin = self.num2date(self.datetime[-1])

            self.qhist = self.ib.reqHistoricalDataEx(
                        contract=self.contract,
                        enddate=dtend, begindate=dtbegin,
                        timeframe=self._timeframe,
                        compression=self._compression,
                        what=self.p.what, useRTH=self.p.useRTH, tz=self._tz,
                        sessionend=self.p.sessionend)

            if self._laststatus != self.DELAYED:
                self.put_notification(self.DELAYED)

            self._state = self._ST_HISTORBACK

            self._statelivereconn = False
            continue  # to reenter the loop and hit st_historback

        if (self.disconnect_marker == 1) and (msg != -1102):
            # reject all RTVolume objects while connection to IB server is not re-established
            print('$$$$$$ connection broken - rejecting all messages')
            return None
        elif (self.disconnect_marker == 1) and (msg == -1102):
            # connection to IB server is operational
            print('$$$$$$ connection restored')
            self.disconnect_marker = 0
                
        if msg is None:  # Conn broken during historical/backfilling
            self.put_notification(self.CONNBROKEN)
            # Try to reconnect
            if not self.ib.reconnect(resub=True):
                self.put_notification(self.DISCONNECTED)
                return False  # failed

            self._statelivereconn = self.p.backfill
            continue
        
        if msg == -354:
            self.put_notification(self.NOTSUBSCRIBED)
            return False
        
        elif msg == -1100:  # conn broken
            # Tell to wait for a message to do a backfill
            # self._state = self._ST_DISCONN
            self._statelivereconn = self.p.backfill
            self.disconnect_marker = 1
            continue
        ....

In the code above I introduced variable self.disconnect_marker which has initial value of 0 (I put this line in __init__ method) and keeps it as long as connection is operational and data is arriving. However, when message with error code -1100 arrives self.disconnect_marker turns to 1 and if some other message, RTVolume for example, arrives later it is discarded and None is returned. Finally, when message with code -1102 (connection reestablished) arrives self.disconnect_marker is given value of 0 again.
It would be very interesting to hear from someone who encountered this issue and tried the above-mentioned method.