3. Unable to open big CSV files
For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Unable to open big CSV files

  • M

    I tried to open CSV file with more than 1 million rows and its write me an error. Than i deleted rows from the same file and it opened it. The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)
    Is there is any way to make it open bigger files?
    Here is the error:

    Starting Portfolio Value: 1000.00
    Traceback (most recent call last):
    File "test.py", line 141, in <module>
    cerebro.run()
    File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
    File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1212, in runstrategies
    data.preload()
    File "C:\Python27\lib\site-packages\backtrader\feed.py", line 685, in preload
    while self.load():
    File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
    _loadret = self._load()
    File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
    return self._loadline(linetokens)
    File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
    self.lines.open[0] = float(next(itoken))
    ValueError: could not convert string to float:

    Same file open with no error when i deleted rows.

    0
  • B
    administrators

    Try running with parameters that would apply to scenarios like live trading. See Docs - Cerebro

    • preload=False (do not preload data feeds)

    • runonce=False (do not precalculate indicators)

    • exactbars=True (reduce all buffers to the minimum necessary for operation)

      This last option is known to have problems when set to the maximum savings (True) under different configurations which resample data feeds or replay several data feeds. If you hit a problem try -2

    @mastr said in Unable to open big CSV files:

    The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)

    The 33 seconds are probably the consequence of continuous allocation and reallocation of the c arrays inside Python when the file is being read.

    0
  • M

    @backtrader said in Unable to open big CSV files:

    preload=False

    Did you mean in "try -2" to this command?
    ('exactbars', -2)
    If so i tried it and all the others and also tried exactbars=true
    now its reaching row number 1,005,370 (reached it twice with exactbars=True and exactbars=-2) and then it write this:

    Traceback (most recent call last):
    File "test.py", line 141, in <module>
    cerebro.run()
    File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
    File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
    File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1541, in _runnext
    drets.append(d.next(ticks=False))
    File "C:\Python27\lib\site-packages\backtrader\feed.py", line 404, in next
    ret = self.load()
    File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
    _loadret = self._load()
    File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
    return self._loadline(linetokens)
    File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
    self.lines.open[0] = float(next(itoken))
    ValueError: could not convert string to float: "11005370

    I also changed the paremeters in both cerbero files the one thats in python27 and the one when test.py is located.

    0