Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Bug using Pandas hdf?

    General Discussion
    3
    4
    3612
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      Chapachan last edited by Chapachan

      Hi, using the standard example for Pandas, but using hdf files, I'm getting an error and I'm not sure if its due to a problem in the data or an error in Backtrader itself:

        File "b2.py", line 69, in <module>
          runstrat()
        File "b2.py", line 46, in runstrat
          cerebro.run()
        File "/Library/Python/2.7/site-packages/backtrader/cerebro.py", line 1127, in run
          runstrat = self.runstrategies(iterstrat)
        File "/Library/Python/2.7/site-packages/backtrader/cerebro.py", line 1209, in runstrategies
          data.preload()
        File "/Library/Python/2.7/site-packages/backtrader/feed.py", line 435, in preload
          while self.load():
        File "/Library/Python/2.7/site-packages/backtrader/feed.py", line 476, in load
          _loadret = self._load()
        File "/Library/Python/2.7/site-packages/backtrader/feeds/pandafeed.py", line 262, in _load
          dt = tstamp.to_pydatetime()
      AttributeError: 'numpy.int64' object has no attribute 'to_pydatetime'
      

      It crashes in Cerebro.run()

      This is the code (basically, copied from example, but using hdf instead of csv). Here goes also the pandas data, just in case it has some error: https://drive.google.com/open?id=0ByTv1orrPKKPYkR2d1NYRHVhSUU

      from __future__ import (absolute_import, division, print_function,
                              unicode_literals)
      
      import argparse
      
      import backtrader as bt
      import backtrader.feeds as btfeeds
      
      import pandas
      
      
      def runstrat():
          args = parse_args()
      
          # Create a cerebro entity
          cerebro = bt.Cerebro(stdstats=False)
      
          # Add a strategy
          cerebro.addstrategy(bt.Strategy)
      
          # Get a pandas dataframe
          datapath = ('datos1.pand')
      
          # Simulate the header row isn't there if noheaders requested
          skiprows = 1 if args.noheaders else 0
          header = None if args.noheaders else 0
      
          dataframe = pandas.read_hdf(datapath,"table",
                                      skiprows=skiprows,
                                      header=header,
                                      parse_dates=True,
                                      index_col=0)
      
          if not args.noprint:
              print('--------------------------------------------------')
              print(dataframe)
              print('--------------------------------------------------')
      
          # Pass it to the backtrader datafeed and add it to the cerebro
          data = bt.feeds.PandasData(dataname=dataframe)
          cerebro.adddata(data)
      
          # Run over everything
          cerebro.run()
      
          # Plot the result
          cerebro.plot(style='bar')
      
      
      def parse_args():
          parser = argparse.ArgumentParser(
              description='Pandas test script')
      
          parser.add_argument('--noheaders', action='store_true', default=False,
                              required=False,
                              help='Do not use header rows')
      
      
          parser.add_argument('--noprint', action='store_true', default=False,
                              help='Print the dataframe')
      
          return parser.parse_args()
      
      
      if __name__ == '__main__':
          runstrat()
      
      

      EDIT: I think the data is correct because it loads correctly when I do print(dataframe):

      --------------------------------------------------
             Timestamp      Open      High       Low     Close      Volume
      0     1483228800  0.008335  0.008377  0.008332  0.008368   22.678130
      1     1483229100  0.008379  0.008412  0.008373  0.008400   23.101541
      2     1483229400  0.008400  0.008463  0.008388  0.008450   42.334102
      ...          ...       ...       ...       ...       ...         ...
      8927  1485906900  0.011050  0.011067  0.011035  0.011046    4.529571
      8928  1485907200  0.011046  0.011091  0.011036  0.011090    9.032207
      
      [8929 rows x 6 columns]
      --------------------------------------------------
      
      

      EDIT2: Ok, I found how to fix this. I don't fully understand why this is needed but now it works. It is needed to transform the timestamp into a python Timestamp object:

      
      def to_datetimeindex(unixtimestamp):
          t=datetime.datetime.fromtimestamp(unixtimestamp)
          return t.strftime("%Y-%m-%d %H:%M:%S")
      
      def to_Timestamp (stringtime):
          return pandas.Timestamp(stringtime)
      
      
      def runstrat():
          args = parse_args()
      
          # Create a cerebro entity
          cerebro = bt.Cerebro(stdstats=False)
      
          # Add a strategy
          cerebro.addstrategy(bt.Strategy)
      
          # Get a pandas dataframe
          datapath = ('datos1.pand')
      
          # Simulate the header row isn't there if noheaders requested
          skiprows = 1 if args.noheaders else 0
          header = None if args.noheaders else 0
      
          dataframe = pandas.read_hdf(datapath,"table",openinterest=-1)
      
          dataframe["Timestamp"] = dataframe["Timestamp"].apply(to_datetimeindex)
          dataframe["Timestamp"] = dataframe["Timestamp"].apply(to_Timestamp)
      
      1 Reply Last reply Reply Quote 0
      • T
        ThatBlokeDave last edited by

        As far as I am aware, BT data feeds require all timestamps / dates to be datetime objects. So that is why you need to convert the TS before adding it.

        In one of my scripts I do this by:

        df.index = pd.to_datetime(df.index, format='%Y-%m-%dT%H:%M:%S.%fZ')
        

        Some more notes here:
        https://www.backtrader.com/docu/pandas-datafeed/pandas-datafeed.html?highlight=pandas

        Another thing you could look at is the datetime keyword argument in the link above. It looks like your timestamp is not the index.

        datetime (default: None)
        None : datetime is the “index” in the Pandas Dataframe
        -1 : autodetect position or case-wise equal name
        = 0 : numeric index to the colum in the pandas dataframe
        string : column name (as index) in the pandas dataframe

        Having said all that, you now have it working! So it does not matter too much :)

        okey linux 1 Reply Last reply Reply Quote 0
        • C
          Chapachan last edited by

          Humm... your one-line solution seems more elegant than mine. Thanks! :)

          1 Reply Last reply Reply Quote 0
          • okey linux
            okey linux @ThatBlokeDave last edited by

            @thatblokedave thanks, that's work for me.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post
            Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors