I have configured a trading calendar that contains daily sessions and I have the problem that the timestamp of the end of session is not being updated after the end of session has been reached.

I have tracked it down to the code where I think a condition is not met which makes it dismiss the end of session condition.

The scenario is like this:

• live paper trading using tick data getting resampled to H1 bars (10:00 to 18:00, 9 bars)
• The trading hours are 9:00 to 17:35
• As said, data is tick-based, but backfilling fills H1 bars (!)

The code I have pinpointed is in resamplefilter.py in function _eoscheck:

        equal = dt == self._nextdteos
        grter = dt > self._nextdteos

        if exact:
            is_eos = equal
        else:
            # if the compared data goes over the endofsession
            # make sure the resampled bar is open and has something before that
            # end of session. It could be a weekend and nothing was delivered
            # until Monday
            if grter:
                is_eos = self.bar.isopen() and self.bar.datetime <= self._nextdteos
            else:
                is_eos = equal

In particular this condition:

                is_eos = self.bar.isopen() and self.bar.datetime <= self._nextdteos

In the moment a backfilling H1 bar with timestamp 18:00 is coming in (and eos being 17:35) then equal is False but grter is True so the if-clause in question is getting reached. The clause will render is_eos to False which in my opinion is the problem which will lead to not considering the session to have ended and therefore not fetching the next following eos timestamp.

The condition returns False since the bar is not open because the 18:00-bar is the first data in the range from 17:00 - to 18:00 (because H1 bars always deliver a single data in each hour). This situation that a single bar opens a bar and goes beyond the end of the session is the core of the problem I guess cause of the bar.isopen condition.

I am tempted to just remove the condition for me and set is_eos to True when grter is True but I feel I don't really understand what the condition is meant to do. I know there even is a multiline comment above it but I still fail to see under which condition the code. Would you give me a hint please regarding that if-condition? Especially what will go wrong when I remove it?

Also in general:
If I do live tick data resampling to H1 bars. Is it ok for the backtrader to still do backfill with H1 bars? I mean the resampler is expecting tick data but gets H1 bars fed in the beginning. But I think the Oanda data feed does the same so I thought it would be ok.

Many thanks for your help!

Actually I am still planning to get back to this at some point but I got constantly distracted by other things in the last months. It is not dead but I can't make any promises.

I have a question regarding backtrader's behavior when encountering a connection loss while live trading:

As shown in the blog (https://www.backtrader.com/blog/posts/2016-06-21-livedata-feed/live-data-feed.html) when the connection gets lost when in live mode then the current bar (which got interrupted) will still be yielded when the bar ends.

This can be seen here (bar 1443):

Output (skipping the initial known part):


...
1440, 2016-06-20T18:16:20.000000, 16.05, 16.05, 16.04, 16.04, 0.0, 0, 16.048
1441, 2016-06-20T18:16:25.000000, 16.05, 16.05, 16.05, 16.05, 0.0, 0, 16.05
***** DATA NOTIF: LIVE
1442, 2016-06-20T18:16:30.000000, 15.9, 15.9, 15.89, 15.9, 11.0, 0, 16.02
***** STORE NOTIF: <error id=-1, errorCode=1100, errorMsg=Connectivity between IB and TWS has been lost.>
***** STORE NOTIF: <error id=-1, errorCode=2105, errorMsg=HMDS data farm connection is broken:demohmds>
***** STORE NOTIF: <error id=-1, errorCode=2103, errorMsg=Market data farm connection is broken:ibdemo>
1443, 2016-06-20T18:16:35.000000, 15.9, 15.9, 15.89, 15.9, 28.0, 0, 15.988
***** STORE NOTIF: <error id=-1, errorCode=1102, errorMsg=Connectivity between IB and TWS has been restored - data maintained.>
***** STORE NOTIF: <error id=-1, errorCode=2106, errorMsg=HMDS data farm connection is OK:demohmds>
***** STORE NOTIF: <error id=-1, errorCode=2104, errorMsg=Market data farm connection is OK:ibdemo>
***** DATA NOTIF: DELAYED
1444, 2016-06-20T18:16:40.000000, 16.04, 16.04, 16.03, 16.04, 0.0, 0, 15.986
1445, 2016-06-20T18:16:45.000000, 16.03, 16.04, 16.03, 16.04, 0.0, 0, 15.986
1446, 2016-06-20T18:16:50.000000, 16.04, 16.04, 16.03, 16.03, 0.0, 0, 15.982
1447, 2016-06-20T18:16:55.000000, 16.04, 16.04, 16.03, 16.04, 0.0, 0, 16.01
1448, 2016-06-20T18:17:00.000000, 16.03, 16.04, 16.03, 16.04, 0.0, 0, 16.038
1449, 2016-06-20T18:17:05.000000, 16.03, 16.04, 16.02, 16.03, 0.0, 0, 16.036
1450, 2016-06-20T18:17:10.000000, 15.9, 15.91, 15.9, 15.91, 3.0, 0, 16.01
***** DATA NOTIF: LIVE
1451, 2016-06-20T18:17:15.000000, 15.92, 15.92, 15.9, 15.92, 9.0, 0, 15.988
1452, 2016-06-20T18:17:20.000000, 15.91, 15.91, 15.89, 15.89, 18.0, 0, 15.958
1453, 2016-06-20T18:17:25.000000, 15.89, 15.92, 15.89, 15.89, 24.0, 0, 15.928
...
The narrative:

After bar 1442, the WLAN interface has been disabled

TWS notifications arrive indicating the situation

Bar 1443 is delivered from the resampler, because the platform had some ticks in between 18:16:30.000000 and 18:16:35.000000

I am referring to the last sentence which describes the behavior I am seeing. So I guess this behavior is intended.

I was surprised about this behavior and thought if it wouldn't be better if backtrader would not yield the bar (which is based on incomplete data) and rather backfill it later upon reconnection like the other missing bars?

I mean in worst case you could be trading D1 bars and only had a single tick from that day but still a whole bar would be yielded based on a single tick?

One point is to make sure to use 64bit version of python.

Ok, no problem, I understand. Knowing your reason is already valuable to me, so thanks for that reply.

So I guess 'no answer' means I am totally on the wrong track here, right?

Thanks for your answer! Sorry I could not get back to this earlier. And sorry but I still have problemts to see how it is intended to work.
My main issues are that you say that day does not carry time information and that it is not needed to check whether day is a trading day.
I wrote some little test cases that are runnable and are intended to demonstrate my issues. Possibly they do not meet some assumptions the function schedule made, I am not sure. I thought it would be easier to discuss directly about source code.

@backtrader said in Trading Calendar possible implementation problems?:

@vbs said in Trading Calendar possible implementation problems?:

while True:
...
if day > closing: # current time over eos
day += ONEDAY
continue

What this code does: if the current day given as param is not a trading day, go for the next. A day has no time (which can be equaled to 00:00:00, but closing contains a time component which should be greater than 00:00:00)

If you pass a Monday and it is an active trading day, the loop won't go to the next day. If you pass a Sunday and it isn't a trading day (which is most likely the case), the loop will go to the next day, i.e.: Monday

So, as you say day has no time I conclude that it is supposed to be an instance of datetime.date? But as far as I can see at runtime DataFeed calls schedule passing a regular datetime.datetime instance so it will have a time component:

        dt = self.lines.datetime[0]
        dtime = num2date(dt)
        ...
        _, nexteos = self._calendar.schedule(dtime, self._tz)

I think this is an important point in understanding the code. My following thoughts are based on the assumption that the function really excepts an instance of datetime.datetime which carries a time component. If this is really not true then the following text can be ignored I think. :)

if the time component of the provided parameter day is after the closing time then it will result in an endless loop as demonstrated here:

import datetime

import backtrader as bt


tradingcal = bt.TradingCalendar(open=datetime.time(hour=12), close=datetime.time(hour=22))

# results in an endless loop cause 22:00:02 is always outside of trading hours
sched = tradingcal.schedule(datetime.datetime(2018, 1, 1, 22, 0, 2))

@vbs said in Trading Calendar possible implementation problems?:

Could you maybe comment on my assumption that the interface of the function schedule is to return the opening and closing time of the session that ends next (instead of the opening/closing for the provided day)?

As explained above. It will return the closing of the given day if its a trading day and move to the next if it isn't.

Ok, but as I read the code I always goes forward exactly one day but I cannot see the code checking if that day is a trading day at all. I cannot understand why it would not be needed to check if that following day is a trading day.

I made a small test script that demonstrates my concern about that nextday. The script is runnable so it can be analyzed what is happening.

import datetime

import backtrader as bt


tradingcal = bt.TradingCalendar(open=datetime.time(hour=12),
                                close=datetime.time(hour=22),
                                earlydays=[(datetime.date(2018, 11, 23), datetime.time(hour=12), datetime.time(hour=20))])

sched = tradingcal.schedule(datetime.datetime(2018, 11, 23, 20, 10, 0))

print(sched)

# should skip saturday and sunday and return monday (26-11-2018)
assert(sched == (datetime.datetime(2018, 11, 26, 12), datetime.datetime(2018, 11, 26, 22)))

It starts searching on a friday but after close time. So it skips to the next day which is saturday. It does not check if saturday is a trading day and returns it which is wrong in my opinion. I expected the function to return the upcoming monday (which is the next trading day).

@vbs said in Trading Calendar possible implementation problems?:

2. I think there is a problem for trading hours that start before UTC midnight ...

2005-05-23 is not greater than 2005-05-23 05:00 UTC (which when comparing carries no timezone payload and is simply 2005-05-23 05:00) so day > closing will not happen

Hm, ok, but it will use the trading hours for the date 2005-05-23 which is wrong in my opinion. Because of the timezone of the exchange the trading hours of 2005-05-22 should be used which could be different than the ones for 2005-05-23.

Also a small test script for demonstration:

import datetime

import backtrader as bt
import pytz

tradingcal = bt.TradingCalendar(open=datetime.time(hour=10),
                                close=datetime.time(hour=16, minute=45),
                                earlydays=[(datetime.date(2018, 11, 20), datetime.time(hour=9), datetime.time(hour=18))],
                                )

sched = tradingcal.schedule(datetime.datetime(2018, 11, 21, 3, 0, 0), pytz.timezone('Pacific/Auckland'))

# the requested timestamp actually was date '2018-11-20' (not 21th) according to exhchange's timezone 'Pacific/Auckland' so it should return trading hours
# for that date. those trading hours differ since they are defined by the earlydays parameter (instead of regular trading hours)
assert(sched == (datetime.datetime(2018, 11, 20, 20), datetime.datetime(2018, 11, 21, 5)))

Could you maybe comment on my assumption that the interface of the function schedule is to return the opening and closing time of the session that ends next (instead of the opening/closing for the provided day)?
It's the important point for me. I can hopefully deal with the rest alone.

I am trying to implement my own class derived from TradingCalenderBase and I am trying for some time to understand the existing implementation TradingCalendar. I have problems implementing the function schedule (and I do not fully understand the implementation in TradingCalendar either).

So either its my bad or maybe there even is a bug in the current implementation. The first is more probable :)
I will just explain what I see and what I do not understand, please feel free to correct me where I am wrong. I do not say my findings are facts.

The current implementation in TradingCalendar is this:

    def schedule(self, day, tz=None):
        '''
        Returns the opening and closing times for the given ``day``. If the
        method is called, the assumption is that ``day`` is an actual trading
        day

        The return value is a tuple with 2 components: opentime, closetime
        '''
        while True:
            dt = day.date()
            try:
                i = self._earlydays.index(dt)
                o, c = self.p.earlydays[i][1:]
            except ValueError:  # not found
                o, c = self.p.open, self.p.close

            closing = datetime.combine(dt, c)
            if tz is not None:
                closing = tz.localize(closing).astimezone(UTC)
                closing = closing.replace(tzinfo=None)

            if day > closing:  # current time over eos
                day += ONEDAY
                continue

            opening = datetime.combine(dt, o)
            if tz is not None:
                opening = tz.localize(opening).astimezone(UTC)
                opening = opening.replace(tzinfo=None)

            return opening, closing

This is what I am struggling with the most:

while True:
...
    if day > closing:  # current time over eos
        day += ONEDAY
        continue

I am not totally sure what this code means to do. But when analyzing where schedule is used (in feed._getnexteos) I guess the whole point of schedule is actually not to return opening and closing time for the date specified by date but to return the session that ends next. This would be in contradiction to the function header saying Returns the opening and closing times for the given day. Is this correct so far?

When thinking about that, some possible problems came to mind:

1. Regarding that while-loop: let's say we have regular trading hours Mon to Fri from 09:00 to 18:00. If the function gets called with day being a random trading day date with its time being 18:05 then I suspect it will run into an endless loop cause day will always be greater than closing. ONEDAY will be added again and again but the actual time will never chance so it will always be greater than the current date's closing time. Is this a bug or did I misinterpret? Maybe there are assumptions this function makes to avoid this scenario to ever happen?
   Suggestion: Do the compare day > closing always with the original datetime value passed to the function (and not with the variable day which got increment by ONEDAY)

2. I think there is a problem for trading hours that start before UTC midnight 00:00 and end after. Let's say opening is 22:00 UTC and closing 06:00 UTC.
   Let's take New Zealand exchange which opens at 22:00 UTC and closes at 05:00 UTC (+12h UTC offset). So when calling schedule with day being e.g. 2005-05-23 04:00 UTC we are actually in NZSX's trading hours (2005-05-22 22:00 UTC to 2005-05-23 05:00 UTC) from date 2005-05-22 (22th !) which should be the expected return value. But the function would start search at the provided date 2005-05-23 and therefore miss the 22th and return then opening and closing times for the 23th which are 2005-05-23 22:00 UTC to 2005-05-24 22:00 UTC. This would be wrong I suspsect.
   Suggestion: Do not start searching exactly from the provided date in day but from the day before (Subtract ONEDAY before starting the search).

3. If this is true day > closing then ONEDAY will be added and then searched again for the next day. But there is no check if the next day is a trading day at all. I think this is missing and can lead to returning trading hours for a non-trading day.
   Suggestion: Instead of adding ONEDAY, call _nextday to jump the actual next trading day.

IBStore is clearly in there, I can see it :P

And you are sure that this does not work now? Well, it has to work

store = backtrader.stores.ibstore.IBStore(port=7497)