How to Turn Bid/Ask Data into BT compatible data?
-
Read our blog post on converting bid ask data to OHLC format.
I'm using DukasCopy FX data, which looks like this:
time,ask,bid,ask_volume,bid_volume 2017-01-01 22:00:20.786000,1.05236,1.05148,750000,750000 2017-01-01 22:00:36.636000,1.05236,1.05153,750000,1500000 2017-01-01 22:01:37.024000,1.05236,1.05153,750000,1500000
Modeling your code example..
# Create a Data Feed data = bt.feeds.GenericCSVData( dataname=args.data, #dtformat='%Y-%m-%d %H:%M:%S', #'tmformat', '%H:%M:%S', #tmformat='%H%M%S', # already the default value datetime=1, # position at default time=-1, # position of time open=3, # position of open high=3, low=3, close=3, volume=5, openinterest=-1, # -1 for not present timeframe=bt.TimeFrame.Ticks)
And im getting this error:
Traceback (most recent call last): File "fx_backtest.py", line 160, in <module> cerebro.run() File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 804, in run data.preload() File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 659, in preload while self.load(): File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 454, in load _loadret = self._load() File "/usr/local/lib/python3.5/dist-packages/backtrader/feed.py", line 681, in _load return self._loadline(linetokens) File "/usr/local/lib/python3.5/dist-packages/backtrader/feeds/csvgeneric.py", line 80, in _loadline dtfield += 'T' + linetokens[self.p.time] IndexError: list index out of range
Ideas?
-
Tried another method
class BidAskCSV(btfeeds.GenericCSVData): linesoverride = True # discard usual OHLC structure # datetime must be present and last lines = ('datetime', 'ask', 'bid' ) # datetime (always 1st) and then the desired order for params = ( # (datetime, 0), # inherited from parent class ('ask', 1), # default field pos 1 ('bid', 2), # default field pos 2 ) data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)
But this one throws a different error:
ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'
Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?
-
I pruned out the nano seconds
time,ask,bid,ask_volume 2017-01-01 22:00:20,1.05236,1.05148,750000 2017-01-01 22:00:36,1.05236,1.05153,750000 2017-01-01 22:01:37,1.05236,1.05153,750000 2017-01-01 22:02:18,1.05236,1.05170,750000 2017-01-01 22:02:30,1.05248,1.05170,1500000 2017-01-01 22:02:36,1.05247,1.05170,1500000
Then tried running via the 2nd method:
data = BidAskCSV(dataname=args.data, dtformat=args.dtformat)
Now I get this error:
AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'
-
@Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:
dtfield += 'T' + linetokens[self.p.time]
That means that even if the code you show above has
time=-1
that's not the actual code being executed. In the actual code in executionself.p.time
is for sure>= 0
datetime=1, # position at default
And that's for sure also wrong, because the timestamp is clearly at position
0
-
@Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:
But this one throws a different error:
ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S'Does this have to do with the fact that Dukas data has nanoseconds and it's causing parsing issues?
It means that the
dtformat
you have passed for parsing is empty hence:''
and that cannot be parsed (it the standard pythondatetime
module is capable of handling strings with nanoseconds is unknown) -
@Taewoo-Kim said in How to Turn Bid/Ask Data into BT compatible data?:
Now I get this error:
AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'close'By overriding the lines, the only remaining lines in the data feed are
lines = ('datetime', 'ask', 'bid' )
. Some part of your code (or if you are passing this to the broker) is trying to accessclose
which is no longer part of the hierarchy. -
To summarize: you should check the actual execution code in your first attempt, where even if you show
time=-1
, something else is actually being executed. The part of the source code where the error happens:if self.p.time >= 0: # add time value and format if it's in a separate field dtfield += 'T' + linetokens[self.p.time]
And this can only happend if the parameter
time
is>= 0
-
Note: The forum uses standard
markdown
for the formatting. Your posts would be a lot more readable (before having been edited) if you either-
Prepend 4 spaces to code/shell output blocks
-
Enclose the blocks with a starting:
``` here is the code block ```
See here: http://commonmark.org/help/ (or use the
COMPOSE ?
button in the top-right corner of the editing window) -
-
@backtrader I apologize for that. Didn't even realize such formatting even existed. Will do from now on.
-
OK i figured it out in case anyone else is having problems with this
I downloaded dukas with with dukascopy downloader on windows. It was padding \n with \r\n.. so when i was running backtrader /python on Linux, it was "adding" new lines .
Simple fix
sudo apt-get install dos2unix dos2unix <your csv file>
Problem fixed
-
@taewoo-kim whats the final script to convert dukascopy data?