Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Unable to open big CSV files

    General Code/Help
    2
    3
    1015
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mastr last edited by

      I tried to open CSV file with more than 1 million rows and its write me an error. Than i deleted rows from the same file and it opened it. The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)
      Is there is any way to make it open bigger files?
      Here is the error:

      Starting Portfolio Value: 1000.00
      Traceback (most recent call last):
      File "test.py", line 141, in <module>
      cerebro.run()
      File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
      runstrat = self.runstrategies(iterstrat)
      File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1212, in runstrategies
      data.preload()
      File "C:\Python27\lib\site-packages\backtrader\feed.py", line 685, in preload
      while self.load():
      File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
      _loadret = self._load()
      File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
      return self._loadline(linetokens)
      File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
      self.lines.open[0] = float(next(itoken))
      ValueError: could not convert string to float:

      Same file open with no error when i deleted rows.

      1 Reply Last reply Reply Quote 0
      • B
        backtrader administrators last edited by

        Try running with parameters that would apply to scenarios like live trading. See Docs - Cerebro

        • preload=False (do not preload data feeds)

        • runonce=False (do not precalculate indicators)

        • exactbars=True (reduce all buffers to the minimum necessary for operation)

          This last option is known to have problems when set to the maximum savings (True) under different configurations which resample data feeds or replay several data feeds. If you hit a problem try -2

        @mastr said in Unable to open big CSV files:

        The limit is between 470K to 460K rows and about 33 seconds ( maybe its limited by time to open the file)

        The 33 seconds are probably the consequence of continuous allocation and reallocation of the c arrays inside Python when the file is being read.

        1 Reply Last reply Reply Quote 0
        • M
          mastr last edited by mastr

          @backtrader said in Unable to open big CSV files:

          preload=False

          Did you mean in "try -2" to this command?
          ('exactbars', -2)
          If so i tried it and all the others and also tried exactbars=true
          now its reaching row number 1,005,370 (reached it twice with exactbars=True and exactbars=-2) and then it write this:

          Traceback (most recent call last):
          File "test.py", line 141, in <module>
          cerebro.run()
          File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1127, in run
          runstrat = self.runstrategies(iterstrat)
          File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies
          self._runnext(runstrats)
          File "C:\Python27\lib\site-packages\backtrader\cerebro.py", line 1541, in _runnext
          drets.append(d.next(ticks=False))
          File "C:\Python27\lib\site-packages\backtrader\feed.py", line 404, in next
          ret = self.load()
          File "C:\Python27\lib\site-packages\backtrader\feed.py", line 476, in load
          _loadret = self._load()
          File "C:\Python27\lib\site-packages\backtrader\feed.py", line 707, in _load
          return self._loadline(linetokens)
          File "C:\Python27\lib\site-packages\backtrader\feeds\btcsv.py", line 52, in _loadline
          self.lines.open[0] = float(next(itoken))
          ValueError: could not convert string to float: "11005370

          I also changed the paremeters in both cerbero files the one thats in python27 and the one when test.py is located.

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          • First post
            Last post
          Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors