How to Turn Bid/Ask Data into BT compatible data?



  • Read our blog post on converting bid ask data to OHLC format.

    I'm using DukasCopy FX data, which looks like this:

    time,ask,bid,ask_volume,bid_volume
    2017-01-01 22:00:20.786000,1.05236,1.05148,750000,750000
    2017-01-01 22:00:36.636000,1.05236,1.05153,750000,1500000
    2017-01-01 22:01:37.024000,1.05236,1.05153,750000,1500000
    

    Modeling your code example..

        # Create a Data Feed
        data = bt.feeds.GenericCSVData(
            dataname=args.data,
            #dtformat='%Y-%m-%d %H:%M:%S',
            #'tmformat', '%H:%M:%S',
            #tmformat='%H%M%S',  # already the default value
            datetime=1,  # position at default
            time=-1,  # position of time
            open=3,  # position of open
            high=3,
            low=3,
            close=3,
            volume=5,
            openinterest=-1,  # -1 for not present
            timeframe=bt.TimeFrame.Ticks)
    

    And im getting this error:

    Traceback (most recent call last):
      File "fx_backtest.py", line 160, in <module>
        cerebro.run()
      File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 804, in run
        data.preload()
      File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 659, in preload
        while self.load():
      File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 454, in load
        _loadret = self._load()
      File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 681, in _load
        return self._loadline(linetokens)
      File "/usr/local/lib/python3.5/dist-packages/backtrader/feeds/csvgeneric.py", line 80, in _loadline
        dtfield += 'T' + linetokens[self.p.time]
    IndexError: list index out of range
    

    Ideas?



  • Tried another method

    class BidAskCSV(btfeeds.GenericCSVData):
        linesoverride = True  # discard usual OHLC structure
        # datetime must be present and last
        lines = ('datetime', 'ask', 'bid' )
        # datetime (always 1st) and then the desired order for
        params = (
            # (datetime, 0), # inherited from parent class
            ('ask', 1),  # default field pos 1
            ('bid', 2),  # default field pos 2
    
    
        )
    
    
    
    data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)
    

    But this one throws a different error:

    ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'
    

    Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?



  • I pruned out the nano seconds

        time,ask,bid,ask_volume
    2017-01-01 22:00:20,1.05236,1.05148,750000
    2017-01-01 22:00:36,1.05236,1.05153,750000
    2017-01-01 22:01:37,1.05236,1.05153,750000
    2017-01-01 22:02:18,1.05236,1.05170,750000
    2017-01-01 22:02:30,1.05248,1.05170,1500000
    2017-01-01 22:02:36,1.05247,1.05170,1500000
    

    Then tried running via the 2nd method:

    data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)
    

    Now I get this error:

    AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'
    

  • administrators

    @Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

    dtfield += 'T' + linetokens[self.p.time]

    That means that even if the code you show above has time=-1 that's not the actual code being executed. In the actual code in execution self.p.time is for sure >= 0

    datetime=1, # position at default

    And that's for sure also wrong, because the timestamp is clearly at position 0


  • administrators

    @Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

    But this one throws a different error:
    ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'

    Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?

    It means that the dtformat you have passed for parsing is empty hence: '' and that cannot be parsed (it the standard python datetime module is capable of handling strings with nanoseconds is unknown)


  • administrators

    @Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

    Now I get this error:
    AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'

    By overriding the lines, the only remaining lines in the data feed are lines = ('datetime', 'ask', 'bid' ). Some part of your code (or if you are passing this to the broker) is trying to access close which is no longer part of the hierarchy.


  • administrators

    To summarize: you should check the actual execution code in your first attempt, where even if you show time=-1, something else is actually being executed. The part of the source code where the error happens:

            if self.p.time >= 0:
                # add time value and format if it's in a separate field
                dtfield += 'T' + linetokens[self.p.time]
    

    And this can only happend if the parameter time is >= 0


  • administrators

    Note: The forum uses standard markdown for the formatting. Your posts would be a lot more readable (before having been edited) if you either

    • Prepend 4 spaces to code/shell output blocks

    • Enclose the blocks with a starting:

          ```
          here is the code block
          ```
      

    See here: http://commonmark.org/help/ (or use the COMPOSE ? button in the top-right corner of the editing window)



  • @backtrader I apologize for that. Didn't even realize such formatting even existed. Will do from now on.



  • OK i figured it out in case anyone else is having problems with this

    I downloaded dukas with with dukascopy downloader on windows. It was padding \n with \r\n.. so when i was running backtrader /python on Linux, it was "adding" new lines .

    Simple fix

    sudo apt-get install dos2unix
    dos2unix <your csv file>
    

    Problem fixed


Log in to reply
 

Looks like your connection to Backtrader Community was lost, please wait while we try to reconnect.