Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    Transactions happen for times that don't exist in source data

    General Code/Help
    3
    8
    355
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      ruiarruda last edited by

      I'm trying to use the Pandas Data Feed to load data and run a strategy. I load it like this:

      import yfinance
      a = yfinance.download(tickers="VGAC",period="max",interval="60m", groupby='ticker', prepost=True)
      data = bt.feeds.PandasData(dataname=a, name='VGAC')
      

      I add a transactions analyzer and run the strategy like this:

      cerebro.addanalyzer(btanalyzers.Transactions, _name = "trans")
      back = cerebro.run()
      

      And I then look at the transactions like this:

      back[0].analyzers.trans.get_analysis()
      

      To my surprise there are many transactions happening with a datetime that isn't in the source data. For instance, a transaction at (datetime.datetime(2020, 11, 25, 19, 30), when that value isn't in the source data (the "a" dataframe) datetime column.

      Does backtrader automatically fill gaps in candles? Or what is happening here? Couldn't find references to this in the documentation. After simplifying my tests by orders of magnitude in an attempt to obtain a result I actually understand and expect, I still can't. Surely it is because I lack in intellectual capacity, as apparently after several years of coding without major hassles, I discover a library that somehow is super easy to use for everyone else but makes me the most lost I've ever been when trying to tackle any problem, ever.

      run-out 1 Reply Last reply Reply Quote 0
      • run-out
        run-out @ruiarruda last edited by

        @ruiarruda Could you include the entire code you are running and we'll sort it out with you.

        RunBacktest.com

        R 1 Reply Last reply Reply Quote 1
        • R
          ruiarruda @run-out last edited by

          @run-out thank you. I couldn't reproduce the problem again (that's how lost I am), but in my attempts I realized some things:

          • An indicator (RSI) I still had from the template code I had copy-pasted is influencing the strategy even though I don't use it anywhere after declaring it, and in fact I can't even run the code if I comment it out. I get the message "ValueError: min() arg is an empty sequence"

          • I'm trying to trade when the price hits a 3-hour low, placing a bracket order with that price as the buy price, and a take profit 0.8% higher and a stop loss 5% lower. That buy signal of mine apparently only starts being searched for after the period of the RSI is over... Couldn't be more lost

          • The order notification (and the transaction analyzer, sometimes) shows <backtrader.linebuffer.LineBuffer object at ...> instead of the price for the buy order only, showing a decimal number, as expected, for the other ones

          • Also, that price of the buy order, that I can't see as I explained above, is apparently set as the low price of the NEXT candle after the buy signal is found (I expected it to be set to the current candle's low and only buy if the price is met on the next candle)

          I'm rather stunned at how little sense I can make of anything that is happening. Here is the full code:

          import backtrader as bt
          from datetime import datetime
          import backtrader.analyzers as btanalyzers
          import yfinance
          
          class firstStrategy(bt.Strategy):
          
              def __init__(self):
                  self.min_profit_sell = 0.008
                  self.stop_loss = .05
                  self.rsi = bt.indicators.RSI_SMA(self.data.close, period=14)
                  self.orefs = []
          
              def notify_order(self, order):
                  print('{}: Order ref: {} / Type {} / Status {} / Size {} / Price {}'.format(
                      self.data.datetime.datetime(0),
                      order.ref, 'Buy' * order.isbuy() or 'Sell',
                      order.getstatusname(),
                      round(order.size, 2),
                      'NA' if not order.price else order.price))
          
                  if order.status == order.Completed:
                      self.holdstart = len(self)
          
                  if not order.alive() and order.ref in self.orefs:
                      self.orefs.remove(order.ref)
          
                      
              def next(self):
                  self.interval_low = min(self.data.low.get(size=3))
                  print (self.data.low.get(size=1))
                  if not self.position:
                      if self.data.low == self.interval_low:
                          limitprice = self.data.low * (1 + self.min_profit_sell)
                          print (f'Limit buy price: {self.data.low.get(size=1)[0]}')
                          print (f'Take profit price: {limitprice}')
                          stopprice = self.data.low * (1 - self.stop_loss)
                          print (f'Stop loss price: {stopprice}')
                          os = self.buy_bracket(size=100, limitprice=limitprice, price=self.data.low, stopprice=stopprice)
                          self.orefs = [o.ref for o in os]
          
          #Variable for our starting cash
          startcash = 10000
          
          #Create an instance of cerebro
          cerebro = bt.Cerebro()
          
          #Add our strategy
          cerebro.addstrategy(firstStrategy)
          
          a = yfinance.download(tickers="VGAC",period="max",interval="60m", groupby='ticker', prepost=True)
          data = bt.feeds.PandasData(dataname=a, name='VGAC')
          
          #Add the data to Cerebro
          cerebro.adddata(data)
          
          # Set our desired cash start
          cerebro.broker.setcash(startcash)
          
          # save a CSV with results
          cerebro.addwriter(bt.WriterFile, csv=True, out='backtest.csv')
          
          cerebro.addanalyzer(btanalyzers.Transactions, _name = "trans")
          cerebro.addanalyzer(btanalyzers.TradeAnalyzer, _name = "trades")
          
          # Run over everything
          back = cerebro.run()
          
          #Get final portfolio Value
          portvalue = cerebro.broker.getvalue()
          pnl = portvalue - startcash
          

          Thanks a lot to you or anyone who can point me in the right direction to not give up on this library before even really starting

          run-out 1 Reply Last reply Reply Quote 0
          • run-out
            run-out @ruiarruda last edited by

            @ruiarruda Two major issues and I modified some minor code.

            1. Your code was inputting the entire line into the bracket order for price, you need to use square brackets.
            price=self.data.low[0],
            
            1. The next issue you had was you were using a very very tight limit price, which basically means you are going to sell almost immediately due to volatility. I adjusted it to:
            self.min_profit_sell = 0.05
            

            Here is the total code.

            import backtrader as bt
            from datetime import datetime
            import backtrader.analyzers as btanalyzers
            import yfinance
            
            
            class firstStrategy(bt.Strategy):
                def __init__(self):
                    self.min_profit_sell = 0.05
                    self.stop_loss = 0.05
                    self.rsi = bt.indicators.RSI_SMA(self.data.close, period=14)
                    self.orefs = []
            
                def log(self, txt, dt=None):
                    """ Logging function fot this strategy"""
                    dt = dt or self.data.datetime[0]
                    if isinstance(dt, float):
                        dt = bt.num2date(dt)
                    print("%s, %s" % (dt.date(), txt))
            
                def notify_order(self, order):
                    print(
                        "{}: Order ref: {} / Type {} / Status {} / Size {} / Price {:5.4f}".format(
                            self.data.datetime.datetime(0),
                            order.ref,
                            "Buy" * order.isbuy() or "Sell",
                            order.getstatusname(),
                            round(order.size, 2),
                            order.created.price,
                        )
                    )
            
                    if order.status == order.Completed:
                        self.holdstart = len(self)
            
                    if not order.alive() and order.ref in self.orefs:
                        self.orefs.remove(order.ref)
            
                def next(self):
                    self.interval_low = min(self.data.low.get(size=3))
                    self.log(
                        f"Open: {self.data.open[0]:5.2f}, High: {self.data.high[0]:5.2f}, "
                        f"Low: {self.data.low[0]:5.2f}, Close: {self.data.close[0]:5.2f} "
                    )
                    if not self.position:
                        if self.data.low == self.interval_low:
                            limitprice = self.data.low * (1 + self.min_profit_sell)
                            # print(f'Limit buy price: {self.data.low.get(size=1)[0]}')
                            stopprice = self.data.low * (1 - self.stop_loss)
                            self.log(
                                f"Take profit price: {limitprice:5.2f} "
                                f"Stop loss price: {stopprice:5.2f}"
                            )
            
                            os = self.buy_bracket(
                                size=100,
                                limitprice=limitprice,
                                price=self.data.low[0],
                                stopprice=stopprice,
                            )
                            self.orefs = [o.ref for o in os]
            
            
            # Variable for our starting cash
            startcash = 10000
            
            # Create an instance of cerebro
            cerebro = bt.Cerebro()
            
            # Add our strategy
            cerebro.addstrategy(firstStrategy)
            
            a = yfinance.download(
                tickers="VGAC", period="max", interval="60m", groupby="ticker", prepost=True
            )
            data = bt.feeds.PandasData(dataname=a, name="VGAC")
            
            # Add the data to Cerebro
            cerebro.adddata(data)
            
            # Set our desired cash start
            cerebro.broker.setcash(startcash)
            
            # save a CSV with results
            cerebro.addwriter(bt.WriterFile, csv=True, out="backtest.csv")
            
            cerebro.addanalyzer(btanalyzers.Transactions, _name="trans")
            cerebro.addanalyzer(btanalyzers.TradeAnalyzer, _name="trades")
            
            # Run over everything
            back = cerebro.run()
            
            # Get final portfolio Value
            portvalue = cerebro.broker.getvalue()
            pnl = portvalue - startcash
            
            

            Here is the first printouts...

            2020-11-24, Open:  9.86, High:  9.95, Low:  9.84, Close:  9.91 
            2020-11-24, Take profit price: 10.33 Stop loss price:  9.35
            2020-11-24 20:30:00: Order ref: 1 / Type Buy / Status Submitted / Size 100 / Price 9.8400
            2020-11-24 20:30:00: Order ref: 2 / Type Sell / Status Submitted / Size -100 / Price 9.3480
            2020-11-24 20:30:00: Order ref: 3 / Type Sell / Status Submitted / Size -100 / Price 10.3320
            2020-11-24 20:30:00: Order ref: 1 / Type Buy / Status Accepted / Size 100 / Price 9.8400
            2020-11-24 20:30:00: Order ref: 2 / Type Sell / Status Accepted / Size -100 / Price 9.3480
            2020-11-24 20:30:00: Order ref: 3 / Type Sell / Status Accepted / Size -100 / Price 10.3320
            2020-11-24, Open:  9.86, High:  9.86, Low:  9.86, Close:  9.86 
            2020-11-24, Open:  9.89, High:  9.89, Low:  9.89, Close:  9.89 
            2020-11-25, Open:  9.95, High:  9.95, Low:  9.95, Close:  9.95 
            2020-11-25, Open:  9.90, High:  9.90, Low:  9.85, Close:  9.89 
            2020-11-25, Take profit price: 10.34 Stop loss price:  9.36
            2020-11-25 14:30:00: Order ref: 4 / Type Buy / Status Submitted / Size 100 / Price 9.8500
            2020-11-25 14:30:00: Order ref: 5 / Type Sell / Status Submitted / Size -100 / Price 9.3575
            2020-11-25 14:30:00: Order ref: 6 / Type Sell / Status Submitted / Size -100 / Price 10.3425
            2020-11-25 14:30:00: Order ref: 4 / Type Buy / Status Accepted / Size 100 / Price 9.8500
            2020-11-25 14:30:00: Order ref: 5 / Type Sell / Status Accepted / Size -100 / Price 9.3575
            2020-11-25 14:30:00: Order ref: 6 / Type Sell / Status Accepted / Size -100 / Price 10.3425
            2020-11-25 14:30:00: Order ref: 1 / Type Buy / Status Completed / Size 100 / Price 9.8400
            2020-11-25 14:30:00: Order ref: 4 / Type Buy / Status Completed / Size 100 / Price 9.8500
            2020-11-25, Open:  9.84, High:  9.94, Low:  9.81, Close:  9.85 
            2020-11-25, Open:  9.81, High:  9.99, Low:  9.81, Close:  9.90 
            2020-11-25 16:30:00: Order ref: 3 / Type Sell / Status Completed / Size -100 / Price 10.3320
            2020-11-25 16:30:00: Order ref: 2 / Type Sell / Status Canceled / Size -100 / Price 9.3480
            2020-11-25 16:30:00: Order ref: 6 / Type Sell / Status Completed / Size -100 / Price 10.3425
            2020-11-25 16:30:00: Order ref: 5 / Type Sell / Status Canceled / Size -100 / Price 9.3575
            

            RunBacktest.com

            R 1 Reply Last reply Reply Quote 2
            • R
              ruiarruda @run-out last edited by

              @run-out Thanks a lot for your code. Was great to learn that I was setting the buy price incorrectly.

              However, the most confusing issue still stands: the RSI indicator, which I initialize but do NOT use (intentionally, at least), is somehow necessary for the code to run (if you comment it out on your code, the code breaks). Also, the script waits for the RSI period (14) to be over before starting to search for the buy signal. I don't understand it...

              Regarding the take profit limit price of .8%... I don't get why it would be a problem. In fact it leads to a higher p&l than 5% on this very code. Also, I ran a live bot for some time with that take profit and it was profitable, and I selected it after running my own simple backtest on different options for the price. Can I not trust Backtrader backtesting results if I use .8% as the limit price?

              D 1 Reply Last reply Reply Quote 0
              • D
                davidavr @ruiarruda last edited by

                @ruiarruda said in Transactions happen for times that don't exist in source data:

                However, the most confusing issue still stands: the RSI indicator, which I initialize but do NOT use (intentionally, at least), is somehow necessary for the code to run (if you comment it out on your code, the code breaks).

                Here is the issue

                self.interval_low = min(self.data.low.get(size=3))
                

                In next you are trying to access the last 3 low values but in the first call to next, there is only one value in self.data.low at this point. When you leave in the line for the RSI indicator, BT buffers the first 14 data points before calling next so the RSI indicator will be available (whether you access it or not).

                You could change the offending line to:

                self.interval_low = min(self.data.low.get(size=min(3, len(self.data.low))))
                

                or just create some indicator with a minimum period of 3 to work around this.

                run-out R 2 Replies Last reply Reply Quote 3
                • run-out
                  run-out @davidavr last edited by

                  @davidavr said in Transactions happen for times that don't exist in source data:

                  ))

                  @davidavr Nice find.

                  @ruiarruda Have a look here to add minimum periods.

                  RunBacktest.com

                  1 Reply Last reply Reply Quote 1
                  • R
                    ruiarruda @davidavr last edited by

                    @davidavr thanks a lot, the problem wasn't so complicated afterall

                    @run-out thank you for the pointer, I'll consult it

                    1 Reply Last reply Reply Quote 0
                    • 1 / 1
                    • First post
                      Last post
                    Copyright © 2016, 2017, 2018, 2019, 2020, 2021 NodeBB Forums | Contributors