Unable to open big CSV files
-
I tried to open CSV file with more than 1 million rows and its write me an error. Than i deleted rows from the same file and it opened it. The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)
Is there is any way to make it open bigger files?
Here is the error:Starting Portfolio Value: 1000.00
Traceback (most recent call last):
File "test.py", line 141, in <module>
cerebro.run()
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1212, in runstrategies
data.preload()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 685, in preload
while self.load():
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
_loadret = self._load()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
return self._loadline(linetokens)
File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
self.lines.open[0] = float(next(itoken))
ValueError: could not convert string to float:Same file open with no error when i deleted rows.
-
Try running with parameters that would apply to scenarios like live trading. See Docs - Cerebro
-
preload=False
(do not preload data feeds) -
runonce=False
(do not precalculate indicators) -
exactbars=True
(reduce all buffers to the minimum necessary for operation)This last option is known to have problems when set to the maximum savings (
True
) under different configurations which resample data feeds or replay several data feeds. If you hit a problem try-2
@mastr said in Unable to open big CSV files:
The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)
The 33 seconds are probably the consequence of continuous allocation and reallocation of the
c
arrays inside Python when the file is being read. -
-
@backtrader said in Unable to open big CSV files:
preload=False
Did you mean in "try -2" to this command?
('exactbars', -2)
If so i tried it and all the others and also tried exactbars=true
now its reaching row number 1,005,370 (reached it twice with exactbars=True and exactbars=-2) and then it write this:Traceback (most recent call last):
File "test.py", line 141, in <module>
cerebro.run()
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
self._runnext(runstrats)
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1541, in _runnext
drets.append(d.next(ticks=False))
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 404, in next
ret = self.load()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
_loadret = self._load()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
return self._loadline(linetokens)
File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
self.lines.open[0] = float(next(itoken))
ValueError: could not convert string to float: "11005370I also changed the paremeters in both cerbero files the one thats in python27 and the one when test.py is located.