Trading Calendar possible implementation problems?

vbs

I am trying to implement my own class derived from TradingCalenderBase and I am trying for some time to understand the existing implementation TradingCalendar. I have problems implementing the function schedule (and I do not fully understand the implementation in TradingCalendar either).

So either its my bad or maybe there even is a bug in the current implementation. The first is more probable :)
I will just explain what I see and what I do not understand, please feel free to correct me where I am wrong. I do not say my findings are facts.

The current implementation in TradingCalendar is this:

    def schedule(self, day, tz=None):
        '''
        Returns the opening and closing times for the given ``day``. If the
        method is called, the assumption is that ``day`` is an actual trading
        day

        The return value is a tuple with 2 components: opentime, closetime
        '''
        while True:
            dt = day.date()
            try:
                i = self._earlydays.index(dt)
                o, c = self.p.earlydays[i][1:]
            except ValueError:  # not found
                o, c = self.p.open, self.p.close

            closing = datetime.combine(dt, c)
            if tz is not None:
                closing = tz.localize(closing).astimezone(UTC)
                closing = closing.replace(tzinfo=None)

            if day > closing:  # current time over eos
                day += ONEDAY
                continue

            opening = datetime.combine(dt, o)
            if tz is not None:
                opening = tz.localize(opening).astimezone(UTC)
                opening = opening.replace(tzinfo=None)

            return opening, closing

This is what I am struggling with the most:

while True:
...
    if day > closing:  # current time over eos
        day += ONEDAY
        continue

I am not totally sure what this code means to do. But when analyzing where schedule is used (in feed._getnexteos) I guess the whole point of schedule is actually not to return opening and closing time for the date specified by date but to return the session that ends next. This would be in contradiction to the function header saying Returns the opening and closing times for the given day. Is this correct so far?

When thinking about that, some possible problems came to mind:

Regarding that while-loop: let's say we have regular trading hours Mon to Fri from 09:00 to 18:00. If the function gets called with day being a random trading day date with its time being 18:05 then I suspect it will run into an endless loop cause day will always be greater than closing. ONEDAY will be added again and again but the actual time will never chance so it will always be greater than the current date's closing time. Is this a bug or did I misinterpret? Maybe there are assumptions this function makes to avoid this scenario to ever happen?
Suggestion: Do the compare day > closing always with the original datetime value passed to the function (and not with the variable day which got increment by ONEDAY)
I think there is a problem for trading hours that start before UTC midnight 00:00 and end after. Let's say opening is 22:00 UTC and closing 06:00 UTC.
Let's take New Zealand exchange which opens at 22:00 UTC and closes at 05:00 UTC (+12h UTC offset). So when calling schedule with day being e.g. 2005-05-23 04:00 UTC we are actually in NZSX's trading hours (2005-05-22 22:00 UTC to 2005-05-23 05:00 UTC) from date 2005-05-22 (22th !) which should be the expected return value. But the function would start search at the provided date 2005-05-23 and therefore miss the 22th and return then opening and closing times for the 23th which are 2005-05-23 22:00 UTC to 2005-05-24 22:00 UTC. This would be wrong I suspsect.
Suggestion: Do not start searching exactly from the provided date in day but from the day before (Subtract ONEDAY before starting the search).
If this is true day > closing then ONEDAY will be added and then searched again for the next day. But there is no check if the next day is a trading day at all. I think this is missing and can lead to returning trading hours for a non-trading day.
Suggestion: Instead of adding ONEDAY, call _nextday to jump the actual next trading day.

vbs

Could you maybe comment on my assumption that the interface of the function schedule is to return the opening and closing time of the session that ends next (instead of the opening/closing for the provided day)?
It's the important point for me. I can hopefully deal with the rest alone.

backtrader

@vbs said in Trading Calendar possible implementation problems?:

while True:
...
if day > closing: # current time over eos
day += ONEDAY
continue

What this code does: if the current day given as param is not a trading day, go for the next. A day has no time (which can be equaled to 00:00:00, but closing contains a time component which should be greater than 00:00:00)

If you pass a Monday and it is an active trading day, the loop won't go to the next day. If you pass a Sunday and it isn't a trading day (which is most likely the case), the loop will go to the next day, i.e.: Monday

@vbs said in Trading Calendar possible implementation problems?:

Could you maybe comment on my assumption that the interface of the function schedule is to return the opening and closing time of the session that ends next (instead of the opening/closing for the provided day)?

As explained above. It will return the closing of the given day if its a trading day and move to the next if it isn't.

@vbs said in Trading Calendar possible implementation problems?:

I think there is a problem for trading hours that start before UTC midnight ...

2005-05-23 is not greater than 2005-05-23 05:00 UTC (which when comparing carries no timezone payload and is simply 2005-05-23 05:00) so day > closing will not happen

@vbs said in Trading Calendar possible implementation problems?:

If this is true day > closing then ONEDAY will be added and then searched again for the next day. But there is no check if the next day is a trading day at all. I think this is missing and can lead to returning trading hours for a non-trading day.

If day has been found in the database, it won't go over. This is for the case in which the day is not found and the default times from the parameters are taken. In that case there can be no check to see if the day is a trading day or not, because it wasn't found and one works blindly.

vbs

Thanks for your answer! Sorry I could not get back to this earlier. And sorry but I still have problemts to see how it is intended to work.
My main issues are that you say that day does not carry time information and that it is not needed to check whether day is a trading day.
I wrote some little test cases that are runnable and are intended to demonstrate my issues. Possibly they do not meet some assumptions the function schedule made, I am not sure. I thought it would be easier to discuss directly about source code.

@backtrader said in Trading Calendar possible implementation problems?:

@vbs said in Trading Calendar possible implementation problems?:

while True:
...
if day > closing: # current time over eos
day += ONEDAY
continue

What this code does: if the current day given as param is not a trading day, go for the next. A day has no time (which can be equaled to 00:00:00, but closing contains a time component which should be greater than 00:00:00)

If you pass a Monday and it is an active trading day, the loop won't go to the next day. If you pass a Sunday and it isn't a trading day (which is most likely the case), the loop will go to the next day, i.e.: Monday

So, as you say day has no time I conclude that it is supposed to be an instance of datetime.date? But as far as I can see at runtime DataFeed calls schedule passing a regular datetime.datetime instance so it will have a time component:

        dt = self.lines.datetime[0]
        dtime = num2date(dt)
        ...
        _, nexteos = self._calendar.schedule(dtime, self._tz)

I think this is an important point in understanding the code. My following thoughts are based on the assumption that the function really excepts an instance of datetime.datetime which carries a time component. If this is really not true then the following text can be ignored I think. :)

if the time component of the provided parameter day is after the closing time then it will result in an endless loop as demonstrated here:

import datetime

import backtrader as bt


tradingcal = bt.TradingCalendar(open=datetime.time(hour=12), close=datetime.time(hour=22))

# results in an endless loop cause 22:00:02 is always outside of trading hours
sched = tradingcal.schedule(datetime.datetime(2018, 1, 1, 22, 0, 2))

@vbs said in Trading Calendar possible implementation problems?:

Could you maybe comment on my assumption that the interface of the function schedule is to return the opening and closing time of the session that ends next (instead of the opening/closing for the provided day)?

As explained above. It will return the closing of the given day if its a trading day and move to the next if it isn't.

Ok, but as I read the code I always goes forward exactly one day but I cannot see the code checking if that day is a trading day at all. I cannot understand why it would not be needed to check if that following day is a trading day.

I made a small test script that demonstrates my concern about that nextday. The script is runnable so it can be analyzed what is happening.

import datetime

import backtrader as bt


tradingcal = bt.TradingCalendar(open=datetime.time(hour=12),
                                close=datetime.time(hour=22),
                                earlydays=[(datetime.date(2018, 11, 23), datetime.time(hour=12), datetime.time(hour=20))])

sched = tradingcal.schedule(datetime.datetime(2018, 11, 23, 20, 10, 0))

print(sched)

# should skip saturday and sunday and return monday (26-11-2018)
assert(sched == (datetime.datetime(2018, 11, 26, 12), datetime.datetime(2018, 11, 26, 22)))

It starts searching on a friday but after close time. So it skips to the next day which is saturday. It does not check if saturday is a trading day and returns it which is wrong in my opinion. I expected the function to return the upcoming monday (which is the next trading day).

@vbs said in Trading Calendar possible implementation problems?:

I think there is a problem for trading hours that start before UTC midnight ...

2005-05-23 is not greater than 2005-05-23 05:00 UTC (which when comparing carries no timezone payload and is simply 2005-05-23 05:00) so day > closing will not happen

Hm, ok, but it will use the trading hours for the date 2005-05-23 which is wrong in my opinion. Because of the timezone of the exchange the trading hours of 2005-05-22 should be used which could be different than the ones for 2005-05-23.

Also a small test script for demonstration:

import datetime

import backtrader as bt
import pytz

tradingcal = bt.TradingCalendar(open=datetime.time(hour=10),
                                close=datetime.time(hour=16, minute=45),
                                earlydays=[(datetime.date(2018, 11, 20), datetime.time(hour=9), datetime.time(hour=18))],
                                )

sched = tradingcal.schedule(datetime.datetime(2018, 11, 21, 3, 0, 0), pytz.timezone('Pacific/Auckland'))

# the requested timestamp actually was date '2018-11-20' (not 21th) according to exhchange's timezone 'Pacific/Auckland' so it should return trading hours
# for that date. those trading hours differ since they are defined by the earlydays parameter (instead of regular trading hours)
assert(sched == (datetime.datetime(2018, 11, 20, 20), datetime.datetime(2018, 11, 21, 5)))

vbs

So I guess 'no answer' means I am totally on the wrong track here, right?

backtrader

It only means that time is scarce and sometimes messages, especially those with complex content, are quickly overrun by simpler things and other-life things

vbs

Ok, no problem, I understand. Knowing your reason is already valuable to me, so thanks for that reply.

Adis Ojeda

Hey @vbs I observed the same error when creating my own calendar; I also feel this code might be erroneous. My data holds price information after the trading session. Once it gets outside of the session, it enters an infinite loop as the time is always after the end of the session.