Unable to open big CSV files

mastr

I tried to open CSV file with more than 1 million rows and its write me an error. Than i deleted rows from the same file and it opened it. The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)
Is there is any way to make it open bigger files?
Here is the error:

Starting Portfolio Value: 1000.00
Traceback (most recent call last):
File "test.py", line 141, in <module>
cerebro.run()
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1212, in runstrategies
data.preload()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 685, in preload
while self.load():
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
_loadret = self._load()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
return self._loadline(linetokens)
File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
self.lines.open[0] = float(next(itoken))
ValueError: could not convert string to float:

Same file open with no error when i deleted rows.

backtrader

Try running with parameters that would apply to scenarios like live trading. See Docs - Cerebro

preload=False (do not preload data feeds)
runonce=False (do not precalculate indicators)
exactbars=True (reduce all buffers to the minimum necessary for operation)

This last option is known to have problems when set to the maximum savings (True) under different configurations which resample data feeds or replay several data feeds. If you hit a problem try -2

@mastr said in Unable to open big CSV files:

The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)

The 33 seconds are probably the consequence of continuous allocation and reallocation of the c arrays inside Python when the file is being read.

mastr

@backtrader said in Unable to open big CSV files:

preload=False

Did you mean in "try -2" to this command?
('exactbars', -2)
If so i tried it and all the others and also tried exactbars=true
now its reaching row number 1,005,370 (reached it twice with exactbars=True and exactbars=-2) and then it write this:

Traceback (most recent call last):
File "test.py", line 141, in <module>
cerebro.run()
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
self._runnext(runstrats)
File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1541, in _runnext
drets.append(d.next(ticks=False))
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 404, in next
ret = self.load()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
_loadret = self._load()
File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
return self._loadline(linetokens)
File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
self.lines.open[0] = float(next(itoken))
ValueError: could not convert string to float: "11005370

I also changed the paremeters in both cerbero files the one thats in python27 and the one when test.py is located.