Resampling tick data - missing candles

hodl

Hi,
First things first: Thank you for the great code! I'm new on boat and already very impressed.

Here's a problem:
I'm trying to import / resample tick data in following format:

1546939820,3997.560000000000,1.000000000000
1546939827,3997.550000000000,0.030602000000
1546939828,3997.550000000000,0.008605000000
1546939828,3999.740000000000,0.608984570000
1546939828,3999.740000000000,0.005633000000
1546939840,4002.930000000000,0.512500000000
1546939840,4003.620000000000,3.748600000000
1546939840,4003.720000000000,2.000000000000
1546939840,4004.470000000000,0.010530000000
1546939840,4005.640000000000,4.000000000000
1546939846,4000.540000000000,0.351588260000
1546939849,3998.220000000000,2.000000000000
1546939850,3997.740000000000,1.306291820000
1546939851,4004.220000000000,1.500000000000
1546939851,4004.870000000000,2.000000000000

my code is:

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import datetime  # For datetime objects
import os.path  # To manage paths
import sys  # To find out the script name (in argv[0])

# Import the backtrader platform
import backtrader as bt
import backtrader.feeds as btfeeds


# Create a Stratey
class TestStrategy(bt.Strategy):

    def log(self, txt, dt=None):
        ''' Logging function for this strategy'''
        dt = dt or self.datas[0].datetime.date(0)
        ti = self.datas[0].datetime.time(0)
        print('%s %s, %s' % (dt.isoformat(), ti.isoformat(), txt))

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.open = self.datas[0].open
        self.high = self.datas[0].high
        self.low = self.datas[0].low
        self.close = self.datas[0].close
        self.volume = self.datas[0].volume

    def next(self):
        # Simply log the closing price of the series from the reference
        self.log('Open, %.2f' % self.open[0])
        self.log('Close, %.2f' % self.close[0])
        self.log('____________')




if __name__ == '__main__':
    # Create a cerebro entity
    cerebro = bt.Cerebro()

    # Add a strategy
    cerebro.addstrategy(TestStrategy)

    # Datas are in a subfolder of the samples. Need to find where the script is
    # because it could have been called from anywhere
    modpath = '../'

    #    datapath = os.path.join(modpath, './datas/orcl-1995-2014.txt')
    datapath = os.path.join(modpath, './datas/bitstampUSDsmall.csv')

    data = btfeeds.GenericCSVData(
        dataname=datapath,
        dtformat=1,
        timeframe=bt.TimeFrame.Ticks,
        #        fromdate=datetime.datetime(2019, 1, 10),
        openinterest=-1,
        open=1,
        high=1,
        low=1,
        close=1,
        volume=2,
        header=False,
        separator=',',
        compression=1,
      )



    # Resample the data
    cerebro.resampledata(
        data,
        timeframe=bt.TimeFrame.Minutes,
        compression=240,
        bar2edge=True,
        adjbartime=True,
        rightedge=True)

    # Add the Data Feed to Cerebro
    cerebro.adddata(data)

    # Set our desired cash start
    cerebro.broker.setcash(100000.0)

    # Print out the starting conditions
    print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

    # Run over everything
    cerebro.run()

    # Print out the final result
    print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

and the result is:

Starting Portfolio Value: 100000.00
2019-01-08 16:00:00, Open, 4031.78
2019-01-08 16:00:00, Close, 4031.82
2019-01-08 16:00:00, ____________
2019-01-08 23:59:59.999989, Open, 4015.90
2019-01-08 23:59:59.999989, Close, 3994.90
2019-01-08 23:59:59.999989, ____________
2019-01-09 08:00:00, Open, 4018.84
2019-01-09 08:00:00, Close, 4019.33
2019-01-09 08:00:00, ____________
2019-01-09 16:00:00, Open, 4025.00
2019-01-09 16:00:00, Close, 3993.42
2019-01-09 16:00:00, ____________
2019-01-09 23:59:59.999989, Open, 4014.37
2019-01-09 23:59:59.999989, Close, 4002.13
2019-01-09 23:59:59.999989, ____________
2019-01-10 08:00:00, Open, 4013.44
2019-01-10 08:00:00, Close, 3803.42
2019-01-10 08:00:00, ____________
2019-01-10 16:00:00, Open, 3791.06
2019-01-10 16:00:00, Close, 3782.40
2019-01-10 16:00:00, ____________
2019-01-11 00:00:00, Open, 3637.34
2019-01-11 00:00:00, Close, 3627.65
2019-01-11 00:00:00, ____________
2019-01-11 08:00:00, Open, 3620.10
2019-01-11 08:00:00, Close, 3643.65
2019-01-11 08:00:00, ____________
2019-01-11 16:00:00, Open, 3639.99
2019-01-11 16:00:00, Close, 3635.20
2019-01-11 16:00:00, ____________
2019-01-12 04:00:00, Open, 3650.40
2019-01-12 04:00:00, Close, 3638.49
2019-01-12 04:00:00, ____________
2019-01-12 12:00:00, Open, 3622.78
2019-01-12 12:00:00, Close, 3617.21
2019-01-12 12:00:00, ____________
2019-01-12 20:00:00, Open, 3629.61
2019-01-12 20:00:00, Close, 3629.11
2019-01-12 20:00:00, ____________
2019-01-13 08:00:00, Open, 3622.47
2019-01-13 08:00:00, Close, 3613.77
2019-01-13 08:00:00, ____________
2019-01-13 16:00:00, Open, 3615.01
2019-01-13 16:00:00, Close, 3611.06
2019-01-13 16:00:00, ____________
2019-01-14 04:00:00, Open, 3502.81
2019-01-14 04:00:00, Close, 3520.01
2019-01-14 04:00:00, ____________
2019-01-14 12:00:00, Open, 3538.24
2019-01-14 12:00:00, Close, 3528.26
2019-01-14 12:00:00, ____________
2019-01-14 20:00:00, Open, 3668.54
2019-01-14 20:00:00, Close, 3683.76
2019-01-14 20:00:00, ____________
2019-01-15 08:00:00, Open, 3668.97
2019-01-15 08:00:00, Close, 3667.96
2019-01-15 08:00:00, ____________
2019-01-15 16:00:00, Open, 3646.76
2019-01-15 16:00:00, Close, 3632.47
2019-01-15 16:00:00, ____________
2019-01-16 00:00:00, Open, 3631.24
2019-01-16 00:00:00, Close, 3576.92
2019-01-16 00:00:00, ____________
2019-01-16 08:00:00, Open, 3597.37
2019-01-16 08:00:00, Close, 3578.60
2019-01-16 08:00:00, ____________
2019-01-16 12:00:00, Open, 3578.60
2019-01-16 12:00:00, Close, 3615.24
2019-01-16 12:00:00, ____________
Final Portfolio Value: 100000.00

Process finished with exit code 0

One can see that resampler skips every other candle except for the last two which are ok.
There is also strange time ending with '59:59.999989' . I tried with different timeframes (minutes, days) and the output is always the same, every second candle is missing.
Anyone has idea what went wrong?
I can provide full sample data (4mb) if needed.

thnx,
T

hodl

Solved.
Resampledata adds data to cerebro automagically. I added it manually with cerebro.adddata().

thnx anyway,
T.

backtrader

@hodl said in Resampling tick data - missing candles:

Resampledata adds data to cerebro automagically

There is nothing automagic in resampledata adding the data feed. The documentation says (Docs - Cerebro)

resampledata(dataname, name=None, **kwargs)
Adds a Data Feed to be resampled by the system

If name is not None it will be put into data._name which is meant for decoration/plotting purposes.

Any other kwargs like timeframe, compression, todate which are supported by the resample filter will be passed transparently

hodl

@backtrader thanks for pointing to relevant docs. I know it was you and not some magic :)