Backtrader Community

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    How to Turn Bid/Ask Data into BT compatible data?

    General Code/Help
    3
    11
    3710
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      Taewoo Kim last edited by backtrader

      Read our blog post on converting bid ask data to OHLC format.

      I'm using DukasCopy FX data, which looks like this:

      time,ask,bid,ask_volume,bid_volume
      2017-01-01 22:00:20.786000,1.05236,1.05148,750000,750000
      2017-01-01 22:00:36.636000,1.05236,1.05153,750000,1500000
      2017-01-01 22:01:37.024000,1.05236,1.05153,750000,1500000
      

      Modeling your code example..

          # Create a Data Feed
          data = bt.feeds.GenericCSVData(
              dataname=args.data,
              #dtformat='%Y-%m-%d %H:%M:%S',
              #'tmformat', '%H:%M:%S',
              #tmformat='%H%M%S',  # already the default value
              datetime=1,  # position at default
              time=-1,  # position of time
              open=3,  # position of open
              high=3,
              low=3,
              close=3,
              volume=5,
              openinterest=-1,  # -1 for not present
              timeframe=bt.TimeFrame.Ticks)
      

      And im getting this error:

      Traceback (most recent call last):
        File "fx_backtest.py", line 160, in <module>
          cerebro.run()
        File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 804, in run
          data.preload()
        File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 659, in preload
          while self.load():
        File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 454, in load
          _loadret = self._load()
        File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 681, in _load
          return self._loadline(linetokens)
        File "/usr/local/lib/python3.5/dist-packages/backtrader/feeds/csvgeneric.py", line 80, in _loadline
          dtfield += 'T' + linetokens[self.p.time]
      IndexError: list index out of range
      

      Ideas?

      B 1 Reply Last reply Reply Quote 0
      • T
        Taewoo Kim last edited by backtrader

        Tried another method

        class BidAskCSV(btfeeds.GenericCSVData):
            linesoverride = True  # discard usual OHLC structure
            # datetime must be present and last
            lines = ('datetime', 'ask', 'bid' )
            # datetime (always 1st) and then the desired order for
            params = (
                # (datetime, 0), # inherited from parent class
                ('ask', 1),  # default field pos 1
                ('bid', 2),  # default field pos 2
        
        
            )
        
        
        
        data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)
        

        But this one throws a different error:

        ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'
        

        Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?

        B 1 Reply Last reply Reply Quote 1
        • T
          Taewoo Kim last edited by backtrader

          I pruned out the nano seconds

              time,ask,bid,ask_volume
          2017-01-01 22:00:20,1.05236,1.05148,750000
          2017-01-01 22:00:36,1.05236,1.05153,750000
          2017-01-01 22:01:37,1.05236,1.05153,750000
          2017-01-01 22:02:18,1.05236,1.05170,750000
          2017-01-01 22:02:30,1.05248,1.05170,1500000
          2017-01-01 22:02:36,1.05247,1.05170,1500000
          

          Then tried running via the 2nd method:

          data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)
          

          Now I get this error:

          AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'
          
          B 1 Reply Last reply Reply Quote 0
          • B
            backtrader administrators @Taewoo Kim last edited by

            @Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

            dtfield += 'T' + linetokens[self.p.time]

            That means that even if the code you show above has time=-1 that's not the actual code being executed. In the actual code in execution self.p.time is for sure >= 0

            datetime=1, # position at default

            And that's for sure also wrong, because the timestamp is clearly at position 0

            1 Reply Last reply Reply Quote 0
            • B
              backtrader administrators @Taewoo Kim last edited by

              @Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

              But this one throws a different error:
              ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'

              Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?

              It means that the dtformat you have passed for parsing is empty hence: '' and that cannot be parsed (it the standard python datetime module is capable of handling strings with nanoseconds is unknown)

              1 Reply Last reply Reply Quote 0
              • B
                backtrader administrators @Taewoo Kim last edited by

                @Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

                Now I get this error:
                AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'

                By overriding the lines, the only remaining lines in the data feed are lines = ('datetime', 'ask', 'bid' ). Some part of your code (or if you are passing this to the broker) is trying to access close which is no longer part of the hierarchy.

                1 Reply Last reply Reply Quote 0
                • B
                  backtrader administrators last edited by

                  To summarize: you should check the actual execution code in your first attempt, where even if you show time=-1, something else is actually being executed. The part of the source code where the error happens:

                          if self.p.time >= 0:
                              # add time value and format if it's in a separate field
                              dtfield += 'T' + linetokens[self.p.time]
                  

                  And this can only happend if the parameter time is >= 0

                  1 Reply Last reply Reply Quote 0
                  • B
                    backtrader administrators last edited by backtrader

                    Note: The forum uses standard markdown for the formatting. Your posts would be a lot more readable (before having been edited) if you either

                    • Prepend 4 spaces to code/shell output blocks

                    • Enclose the blocks with a starting:

                          ```
                          here is the code block
                          ```
                      

                    See here: http://commonmark.org/help/ (or use the COMPOSE ? button in the top-right corner of the editing window)

                    T 1 Reply Last reply Reply Quote 0
                    • T
                      Taewoo Kim @backtrader last edited by

                      @backtrader I apologize for that. Didn't even realize such formatting even existed. Will do from now on.

                      1 Reply Last reply Reply Quote 0
                      • T
                        Taewoo Kim last edited by

                        OK i figured it out in case anyone else is having problems with this

                        I downloaded dukas with with dukascopy downloader on windows. It was padding \n with \r\n.. so when i was running backtrader /python on Linux, it was "adding" new lines .

                        Simple fix

                        sudo apt-get install dos2unix
                        dos2unix <your csv file>
                        

                        Problem fixed

                        Михаил Е.К. 1 Reply Last reply Reply Quote 0
                        • Михаил Е.К.
                          Михаил Е.К. @Taewoo Kim last edited by

                          @taewoo-kim whats the final script to convert dukascopy data?

                          1 Reply Last reply Reply Quote 0
                          • 1 / 1
                          • First post
                            Last post
                          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors