For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Use datetime column as bar open for Pandas Datafeed



  • Hello,

    I have a pandas dataframe which I want to use for backfilling before LIVE trading. The dataframe looks like this as csv:

    datetime;open;high;low;close
    2018-01-29 01:15:00;0.9986;0.99904;0.9986;0.99884
    2018-01-29 01:15:30;0.99884;0.99898;0.99884;0.998965
    

    The dataframe's datetime column is the open time of the bar. As far as I know backtrader assumes the datetime of the bar as the close time, how can I make sure that the datetime column is read as open time? This is crucial for backfilling operation.


  • administrators

    A bar has a single timestamp. It is neither close nor open. What you mean is that the natural assumption is that because you see 4 components (namely OHLC), the timestamp corresponds to the latest of those components which is the C (aka close) The code in backtrader assumes nothing, people do.

    Your timestamps seem to be 30 seconds apart. Your best bet: remove 30 seconds from each timestamp and you will have your assumed open timestamp.



  • thank you for the explanation.

    I want to use this df as backfill_start parameter for IB LIVE and do the following:

    hist_data = bt.feeds.PandasData(dataname=df, timeframe=bt.TimeFrame.Seconds, compression=30, open=0, high=1, low=2, close=3, volume=None, openinterest=None)
    data = ibstore.getdata(dataname=instrumentName, tz=pytz.timezone( 'UTC'), backfill_from=hist_data)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Seconds, compression=30)
    

    This would first backfill from pandas df and then will start getting delayed data from IB. Dataframe's last bar's timestamp is 2018-01-29 01:15:30, which is the OHLC from 01:15:30 until 01:16:00. What will be the timestamp of the first resampled bar (the first delayed data)? Which timespan will that bar have?

    Thanks


  • administrators

    @bb2 said in Use datetime column as bar open for Pandas Datafeed:

    Which timespan will that bar have?

    bars haven't got a timespan. They have a timestamp.

    @bb2 said in Use datetime column as bar open for Pandas Datafeed:

    What will be the timestamp of the first resampled bar (the first delayed data)?

    That depends on the data you pass. Your info is what the timestamp is for the last bar and not for the 1st.

    Isn't it easier if you run it and you check what the timestamp is?



  • @backtrader said in Use datetime column as bar open for Pandas Datafeed:

    bars haven't got a timespan. They have a timestamp.

    technically they might not have a timespan but in real life they do. So if 01:15:30 is the last timestamp bt reads, what will be the first 30 sec resampled bar's timestamp and which timespan would it have in real life?

    The reason I ask questions here is I believe these could be asked by other people as well. There are also some ambiguous parts of backtrader (like the lag in IB live data retrieval I asked in another thread) for me and I am trying to cover them by asking questions here.


  • administrators

    If you want a bar to have a timespan, then there is nothing which can be done against it. You as a human know that the bar comprises information for a given timespan, but the bar itself has no span, it has a timestamp.

    Please, don't hesitate to write a framework which has bars with two timestamps: opening time and closing time. The bars will then have not only a notional timespan but also a defined one. You may then work out if the timespan is defined as fixed or as variable depending on the actual underlying information which makes up the bar.



  • @backtrader

    No need to get offended, I am just trying to find a solution to my problem here. I think backtrader is capable and there are some missing bits I need to figure out and thats it.

    I do not need a framework which has bars with two timestamps. I just want to find out how backtrader handles backfilling when IB backfilling is combined with another source.

    When bt resamples live IB data, it provides timestamps for the OHLC. That is if I got it correctly the close time of the bar. I understand you technically call it timestamp, but the meaning of that timestamp basically is "this bar is the OHLC for the last 30 seconds from x to bar.datetime". Can you please correct me if I am wrong?


  • administrators

    There is no offense. You are simply obsessed with the fact that this framework works with "timespans" and it doesn't.

    @bb2 said in Use datetime column as bar open for Pandas Datafeed:

    but the meaning of that timestamp basically is "this bar is the OHLC for the last 30 seconds from x to bar.datetime".

    @bb2 said in Use datetime column as bar open for Pandas Datafeed:

    Can you please correct me if I am wrong?

    Yes, you are wrong. The timestamp it's a timestamp and not a timespan and the convention is to consider it is the delivery time of the bar. But you can have your own convention.



  • @backtrader alright. Then let's assume IB will backfill delayed data after a pandas dataframe source and IB source is resampled at 30 seconds.

    If the timestamp of the dataframe's last bar is 01:15:30, then I suppose first delivered delayed bar will have timestamp 01:16:00. Is this correct?

    If the 01:16:00 timestamped bar carries OHLC info of 01:15:30 - 01:16:00, then there will be duplicate data because the last timestamp in the dataframe is 01:15:30 and it is timestamped, for convenience, as the open time of the bar.

    What would you suggest to avoid the duplication? Simply shift datetime column of the pandas dataframe?