How to Turn Bid/Ask Data into BT compatible data?

Taewoo Kim

Read our blog post on converting bid ask data to OHLC format.

I'm using DukasCopy FX data, which looks like this:

time,ask,bid,ask_volume,bid_volume
2017-01-01 22:00:20.786000,1.05236,1.05148,750000,750000
2017-01-01 22:00:36.636000,1.05236,1.05153,750000,1500000
2017-01-01 22:01:37.024000,1.05236,1.05153,750000,1500000

Modeling your code example..

    # Create a Data Feed
    data = bt.feeds.GenericCSVData(
        dataname=args.data,
        #dtformat='%Y-%m-%d %H:%M:%S',
        #'tmformat', '%H:%M:%S',
        #tmformat='%H%M%S',  # already the default value
        datetime=1,  # position at default
        time=-1,  # position of time
        open=3,  # position of open
        high=3,
        low=3,
        close=3,
        volume=5,
        openinterest=-1,  # -1 for not present
        timeframe=bt.TimeFrame.Ticks)

And im getting this error:

Traceback (most recent call last):
  File "fx_backtest.py", line 160, in <module>
    cerebro.run()
  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 804, in run
    data.preload()
  File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 659, in preload
    while self.load():
  File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 454, in load
    _loadret = self._load()
  File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 681, in _load
    return self._loadline(linetokens)
  File "/usr/local/lib/python3.5/dist-packages/backtrader/feeds/csvgeneric.py", line 80, in _loadline
    dtfield += 'T' + linetokens[self.p.time]
IndexError: list index out of range

Ideas?

Taewoo Kim

Tried another method

class BidAskCSV(btfeeds.GenericCSVData):
    linesoverride = True  # discard usual OHLC structure
    # datetime must be present and last
    lines = ('datetime', 'ask', 'bid' )
    # datetime (always 1st) and then the desired order for
    params = (
        # (datetime, 0), # inherited from parent class
        ('ask', 1),  # default field pos 1
        ('bid', 2),  # default field pos 2


    )



data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)

But this one throws a different error:

ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'

Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?

Taewoo Kim

I pruned out the nano seconds

    time,ask,bid,ask_volume
2017-01-01 22:00:20,1.05236,1.05148,750000
2017-01-01 22:00:36,1.05236,1.05153,750000
2017-01-01 22:01:37,1.05236,1.05153,750000
2017-01-01 22:02:18,1.05236,1.05170,750000
2017-01-01 22:02:30,1.05248,1.05170,1500000
2017-01-01 22:02:36,1.05247,1.05170,1500000

Then tried running via the 2nd method:

data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)

Now I get this error:

AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'

backtrader

@Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

dtfield += 'T' + linetokens[self.p.time]

That means that even if the code you show above has time=-1 that's not the actual code being executed. In the actual code in execution self.p.time is for sure >= 0

datetime=1, # position at default

And that's for sure also wrong, because the timestamp is clearly at position 0

backtrader

@Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

But this one throws a different error:
ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'

Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?

It means that the dtformat you have passed for parsing is empty hence: '' and that cannot be parsed (it the standard python datetime module is capable of handling strings with nanoseconds is unknown)

backtrader

@Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:

Now I get this error:
AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'

By overriding the lines, the only remaining lines in the data feed are lines = ('datetime', 'ask', 'bid' ). Some part of your code (or if you are passing this to the broker) is trying to access close which is no longer part of the hierarchy.

backtrader

To summarize: you should check the actual execution code in your first attempt, where even if you show time=-1, something else is actually being executed. The part of the source code where the error happens:

        if self.p.time >= 0:
            # add time value and format if it's in a separate field
            dtfield += 'T' + linetokens[self.p.time]

And this can only happend if the parameter time is >= 0

backtrader

Note: The forum uses standard markdown for the formatting. Your posts would be a lot more readable (before having been edited) if you either

Prepend 4 spaces to code/shell output blocks

Enclose the blocks with a starting:

    ```
    here is the code block
    ```

See here: http://commonmark.org/help/ (or use the COMPOSE ? button in the top-right corner of the editing window)

Taewoo Kim

@backtrader I apologize for that. Didn't even realize such formatting even existed. Will do from now on.

Taewoo Kim

OK i figured it out in case anyone else is having problems with this

I downloaded dukas with with dukascopy downloader on windows. It was padding \n with \r\n.. so when i was running backtrader /python on Linux, it was "adding" new lines .

Simple fix

sudo apt-get install dos2unix
dos2unix <your csv file>

Problem fixed

Михаил Е.К.

@taewoo-kim whats the final script to convert dukascopy data?