Navigation

    Backtrader Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

    it should be a bug. size with decimal not works correctly

    General Code/Help
    3
    19
    122
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • well cao
      well cao last edited by

      i have confirm that is the size with decimal for example 3.5,0.01, 100.382 .etc
      the notify_trade with the trade.justopened can not be fired sometimes.

      look at the topics:
      https://community.backtrader.com/topic/937/notify_trade-not-firing-on-subsequent-trades-when-using-tradeid
      it also used the decimal size

      you can reproduce the issue easily as the step:

      :)set fixed size as any decimal number let's say 0.39
      :)set and record the opened trade's unique id when trade.justopened fired
      :)count the total trade ids after backtest finished using stop function
      and you will see the number of trades you recorded with trade.justopened is not match the number Actually executed

      1 Reply Last reply Reply Quote 0
      • well cao
        well cao last edited by

        https://community.backtrader.com/topic/1369/self-buy-size-1-5-does-not-work/7

        another topic with decimal size.

        since these issues have common point: float size

        so it's not coincidence, i should be a bug. we should make bt work correctly with float size

        1 Reply Last reply Reply Quote 0
        • A
          ab_trader last edited by

          there is no evidence of any bugs in the posts you referred to, just misunderstanding of the bt . and actually in the 2nd post it was an attempt to buy more than allowed by available cash.

          if you think that the bug encountered, please post here the script, logs of the prices, orders and trades with the sizes.

          well cao 2 Replies Last reply Reply Quote 0
          • well cao
            well cao @ab_trader last edited by

            @ab_trader
            it think it's a bug
            ok let code talk

            import backtrader as bt
            import shortuuid
            from addict import Dict
            
            
            class S1(bt.Strategy):
                params = (
                    ('size', 1),
                    ('signals_data', None),
                    ('atr_times_for_break_even', 13),
                    ('atr_times_for_stop_loss', 13),
                    ('atr_times_for_take_profit', 26),
                )
            
                def __init__(self):
                    self.data_index = 0
                    self.signal_id = 0
                    self.my_trades = Dict()
                    self.my_trade_opened = Dict()
                    self.my_trades_closed = Dict()
                    self.not_recorded_by_opened = []
                    if self.p.signals_data is None:
                        raise Exception('signal_data_needed!')
                        pass
                    else:
                        pass
                    pass
            
                def next(self):
                    s_signals_data = self.p.signals_data.iloc[self.data_index]
                    atr_value = s_signals_data.atr
                    if self.data_index > self.data.buflen() - 2:
                        return
                        pass
                    if s_signals_data.trend_turn != 0:
                        last_close_price = self.data.close[-1]
                        # tradeid = shortuuid.uuid()
                        tradeid = self.signal_id
                        if s_signals_data.trend_turn == 1:
                            stop_loss = last_close_price - atr_value * self.p.atr_times_for_stop_loss
                            limit_price = last_close_price + atr_value * self.p.atr_times_for_take_profit
                            orders_list = self.buy_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                                           tradeid=tradeid, exectype=bt.Order.Market)
                            self.signal_id += 1
                            pass
                        elif s_signals_data.trend_turn == -1:
                            stop_loss = last_close_price + atr_value * self.p.atr_times_for_stop_loss
                            limit_price = last_close_price - atr_value * self.p.atr_times_for_take_profit
                            orders_list = self.sell_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                                            tradeid=tradeid, exectype=bt.Order.Market)
                            self.signal_id += 1
                            pass
                        pass
                    self.data_index += 1
            
                def notify_trade(self, trade):
                    if trade.justopened:
                        tradeid = trade.tradeid
                        self.my_trade_opened[tradeid] = tradeid
                        self.my_trades[tradeid] = tradeid
                    if trade.isclosed:
                        tradeid = trade.tradeid
                        self.my_trades_closed[tradeid] = tradeid
                        try:
                            del self.my_trades[trade.tradeid]
                        except Exception:
                            self.not_recorded_by_opened.append(tradeid)
                            print('if there is no bug this should not appear', ' the tradeid ', tradeid,
                                  ' not be recorded by trade.justopened when trade_isclosed fired')
            
                def stop(self):
                    print('*' * 100)
                    print('stop_executed')
                    print('len_of_trade.justopened: ', len(self.my_trade_opened))
                    print('len_of_not_recorded_by_trade.justopened: ', len(self.not_recorded_by_opened))
                    print('view_of_the_dict_recorded_by_trade.justopened: ')
                    print(self.my_trade_opened)
                    pass
            
            

            and the log:

            if there is no bug this should not appear  the tradeid  17  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  18  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  16  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  22  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  25  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  27  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  26  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  24  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  28  not be recorded by trade.justopened when trade_isclosed fired
            if there is no bug this should not appear  the tradeid  21  not be recorded by trade.justopened when trade_isclosed fired
            ****************************************************************************************************
            stop_executed
            len_of_trade.justopened:  39
            len_of_not_recorded_by_trade.justopened:  10
            view_of_the_dict_recorded_by_trade.justopened: 
            {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 19: 19, 20: 20, 23: 23, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48}
            
            1 Reply Last reply Reply Quote 0
            • well cao
              well cao @ab_trader last edited by

              @ab_trader
              above comment is using a fractional size
              if you give the set the size as an integer number everything works correctly!
              the correct log is:

              ****************************************************************************************************
              stop_executed
              len_of_trade.justopened:  49
              len_of_not_recorded_by_trade.justopened:  0
              view_of_the_dict_recorded_by_trade.justopened: 
              {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48}
              

              notice this line

              len_of_not_recorded_by_trade.justopened:  0
              

              that means every trade fired by trade.isclosed is record by trade.justopened

              well cao 1 Reply Last reply Reply Quote 0
              • well cao
                well cao @well cao last edited by

                @ab_trader
                feel free to comment if i havn't make it clearly

                1 Reply Last reply Reply Quote 0
                • A
                  ab_trader last edited by

                  i have nothing to comment here since (1) there is not enough information logged, (2) i can't run this script by myself without significant modification and (3) script is quite complex which may induce additional errors not related to the behavior we are trying to understand. For example, tradeid is internal bt counter and you are passing some unique tradeids with the orders. therefore you may interfere with that internal mechanism.

                  well cao 1 Reply Last reply Reply Quote 0
                  • well cao
                    well cao @ab_trader last edited by

                    @ab_trader
                    thanks for your reply, but from your comment , i don't think you are the owner of this great backtest framework or even a programmer. excuse me if you get offended :)
                    because if you are the owner or contributor to this framework you should know the framework's source code or mechanism.
                    :) the tradeid. check this blog--https://www.backtrader.com/blog/posts/2015-10-05-multitrades/multitrades/
                    the unique tradeid for every trade is for the sake of multiple trades. that's why unique tradeid comes from
                    i don't think this "bug" has a bearing on tradeid. even so it's still a bug.
                    :) after some dig into the source code. i found that some logic based on the size.
                    as we know in computer science if the size is integer for example

                    a=2
                    b=2
                    print(a==2)
                    # the result will be True
                    #but if we use fractional size for example 
                    a=2.0
                    b=2.00000000000000035
                    print(a==b)
                    #the result is False.
                    #but may sometime our "a" variable probaly become the value of "b"
                    

                    just a hint. maybe i'm wrong if so let me know . thanks

                    if you think it's neccessary to i will post my full code version . thank you

                    1 Reply Last reply Reply Quote -1
                    • A
                      ab_trader last edited by

                      @well-cao said in it should be a bug. size with decimal not works correctly:

                      just a hint. maybe i'm wrong if so let me know . thanks
                      if you think it's neccessary to i will post my full code version . thank you

                      if you have any evidence confirming that the behavior observed is a bug, than you are welcome to share corresponding information to community, and maybe even make pull request fixing this bug to https://github.com/backtrader2/backtrader (community version bt).

                      1 Reply Last reply Reply Quote 0
                      • vladisld
                        vladisld last edited by

                        hey @well-cao, could you please provide a full code so we could play with it around ? I'm not sure how to repro using the strategy code above ( for example: what need to be passed as a strategy parameter signals_data ? ). It will be event better if you could use the standard test data sets provided with the backtrader itself - this way it will be easier for us to repro.

                        well cao 1 Reply Last reply Reply Quote 0
                        • well cao
                          well cao @vladisld last edited by

                          @vladisld
                          thanks for your reply
                          here is the full version code, the data read from csv file named "test_data.csv"
                          i will attach it with the code:

                          import backtrader as bt
                          import pandas as pd
                          from addict import Dict  # pip install addict
                          
                          
                          class firstStrategy(bt.Strategy):
                              params = (
                                  ('size', 1),
                                  ('atr_times_for_break_even', 13),
                                  ('atr_times_for_stop_loss', 13),
                                  ('atr_times_for_take_profit', 26),
                              )
                          
                              def __init__(self):
                                  self.bars_index = 0
                                  self.signal_id = 0
                                  self.my_trades = Dict()
                                  self.my_trade_opened = Dict()
                                  self.my_trades_closed = Dict()
                                  self.not_recorded_by_opened = []
                          
                              def next(self):
                                  if self.data.trend_turn[0] != 0:
                                      atr_value = self.data.atr[0]
                                      last_close_price = self.data.close[0]
                                      tradeid = self.signal_id
                                      # tradeid=shortuuid.uuid()
                                      if self.data.trend_turn[0] == 1:
                                          # stop_loss = last_close_price - atr_value * self.p.atr_times_for_stop_loss
                                          stop_loss = last_close_price - atr_value * 13
                                          # limit_price = last_close_price + atr_value * self.p.atr_times_for_take_profit
                                          limit_price = last_close_price + atr_value * 13
                                          orders_list = self.buy_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                                                         tradeid=tradeid, exectype=bt.Order.Market)
                                          # main_side = self.buy(size=self.p.size, exectype=bt.Order.Market, tradeid=tradeid, transmit=False)
                                          # self.sell(size=self.p.size, exectype=bt.Order.Stop, tradeid=tradeid, price=stop_loss, parent=main_side,
                                          #           transmit=False)
                                          # self.sell(size=self.p.size, exectype=bt.Order.Limit, price=limit_price, tradeid=tradeid,
                                          #           parent=main_side, transmit=True)
                          
                                          pass
                                      else:
                                          # stop_loss = last_close_price + atr_value * self.p.atr_times_for_stop_loss
                                          stop_loss = last_close_price + atr_value * 13
                                          # limit_price = last_close_price - atr_value * self.p.atr_times_for_take_profit
                                          limit_price = last_close_price - atr_value * 13
                                          orders_list = self.sell_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                                                          tradeid=tradeid, exectype=bt.Order.Market, price=last_close_price)
                                          # main_side = self.sell(size=self.p.size, exectype=bt.Order.Market, tradeid=tradeid, transmit=False)
                                          # self.buy(size=self.p.size, exectype=bt.Order.Stop, tradeid=tradeid, price=stop_loss, parent=main_side,
                                          #          transmit=False)
                                          # self.buy(size=self.p.size, exectype=bt.Order.Limit, price=limit_price, tradeid=tradeid,
                                          #          parent=main_side, transmit=True)
                          
                                      self.signal_id += 1
                                  pass
                          
                              def notify_trade(self, trade):
                                  if trade.justopened:
                                      tradeid = trade.tradeid
                                      # print('trade_size:', trade.size)
                                      self.my_trade_opened[tradeid] = tradeid
                                      self.my_trades[tradeid] = tradeid
                                  if trade.isclosed:
                                      tradeid = trade.tradeid
                                      # print('closed:----tradeid:', tradeid)
                                      self.my_trades_closed[tradeid] = tradeid
                                      try:
                                          del self.my_trades[trade.tradeid]
                                      except Exception:
                                          self.not_recorded_by_opened.append(tradeid)
                                          print('if there is no bug this should not appear', ' the tradeid ', tradeid,
                                                ' not be recorded by trade.justopened when trade_isclosed fired')
                          
                              def stop(self):
                                  print('*' * 100)
                                  print('stop_executed')
                                  print('len_of_trade.justopened: ', len(self.my_trade_opened))
                                  print('len_of_not_recorded_by_trade.justopened: ', len(self.not_recorded_by_opened))
                                  print('view_of_the_dict_recorded_by_trade.justopened: ')
                                  print(self.my_trade_opened)
                                  pass
                          
                          
                          class MyPandasData(bt.feeds.PandasData):
                              # 添加额外的数据列--extra customized columns
                              lines = (
                                  'trend_turn',
                                  'atr',
                              )
                              params = (
                                  ('datetime', 0),
                                  ('open', -1),
                                  ('high', -1),
                                  ('low', -1),
                                  ('close', -1),
                                  ('volume', 5),
                                  ('openinterest', None),
                                  ('trend_turn', 8),
                                  ('atr', 9),
                              )
                              pass
                          
                          
                          signals_data = pd.read_csv('test_data.csv')
                          signals_data['time'] = pd.to_datetime(signals_data['time'])
                          data = MyPandasData(
                              dataname=signals_data,
                              timeframe=bt.TimeFrame.Minutes,
                          )
                          
                          cerebro = bt.Cerebro()
                          # some size for test below:
                          # fractional size:
                          # size = 0.11687
                          # size = 3.2646498
                          size = 1.3983
                          # integer size
                          # size = 4
                          # size = 3.0  #
                          # size = 3
                          # size = 8
                          cerebro.addstrategy(firstStrategy, size=size)
                          cerebro.adddata(data)
                          startcash = 100000
                          cerebro.broker.setcash(startcash)
                          my_strategy = cerebro.run()[0]
                          # Finally plot the end results
                          # cerebro.plot(style='candlestick')
                          
                          

                          the csv data file----csv file uploaed forbidden. you can download the csv file from here:

                          https://drive.google.com/file/d/1o5M-KaJeQuP9WK8INar3ZUinipVEGHwd/view?usp=sharing

                          1 Reply Last reply Reply Quote 0
                          • vladisld
                            vladisld last edited by

                            It seems that there is indeed a problem with fractional order sizes and trades. I believe the same problem was already briefly mentioned in the following post: https://community.backtrader.com/topic/2840/race-condition-bug-w-orders-and-trades/18

                            Just as a side note:

                            The backtrader author mentioned the support for the fractional order sizes for crypto-currencies trading: https://www.backtrader.com/blog/posts/2019-08-29-fractional-sizes/fractional-sizes/#trading-cryptocurrencies-fractions

                            The support for added in CommissionInfo (getsize method) to return the size of the order given the price and the target percentage. It was intended to be used in order_target_value strategy method as well as to be used by sizers.

                            Back to the problem raised by @well-cao:

                            Debugging an issue a little bit it seems that accumulating a position using fractional orders will hit the floating point representation limits causing the small errors to sneak into the position size. This will in turn cause the position not to be closed properly (leaving the tiny amount of position). Since the trade is only declared closed once the position size is absolute zero - the trade will always stay opened.

                            Simple example (unrelated to backtrader - just python console - here the f is representing the position size):

                            >>> f = 0.0
                            >>> f += 1.3983
                            >>> f
                            1.3983
                            >>> f += 1.3983
                            >>> f
                            2.7966
                            >>> f += 1.3983
                            >>> f
                            4.1949000000000005
                            >>> f -= 1.3983
                            >>> f
                            2.7966000000000006
                            >>> f -= 1.3983
                            >>> f
                            1.3983000000000005
                            >>> f -= 1.3983
                            >>> f
                            4.440892098500626e-16
                            

                            I'm not sure what the right fix should be though and will appreciate any further ideas. ( I can share my debug session and data if needed )

                            well cao 1 Reply Last reply Reply Quote 1
                            • well cao
                              well cao @vladisld last edited by

                              @vladisld @ab_trader
                              thanks for your kind reply.
                              i have noticed that article before - https://www.backtrader.com/blog/posts/2019-08-29-fractional-sizes/fractional-sizes/#trading-cryptocurrencies-fractions
                              if not i will think it's my fault because as the document describe the size should be integer

                               Args:
                                          order: the order object which has (completely or partially)
                                              generated this update
                                          size (int): amount to update the order
                                              if size has the same sign as the current trade a
                              

                              because of that article mentioned above so i think it maybe a bug with fractional size since we allow fractional size to use.

                              that is nice hint for the f example. that 's where i give my doubt where the problem comes from. i had try to fix it but with no luck.
                              any opinion , pull request,or patch to work around is appriceated. thanks

                              well cao 1 Reply Last reply Reply Quote 0
                              • well cao
                                well cao @well cao last edited by

                                @vladisld @ab_trader
                                after some debug.
                                i got it work by modify some source code.
                                the file is "/backtrader/position.py"
                                first import decimal module

                                from decimal import Decimal #add this line first
                                

                                and then

                                modify the update function---near line number: 165
                                replace this line self.size += size with
                                self.size = float(Decimal(str(self.size))+Decimal(str(size)))

                                and so far it works as expected. if i'm right maybe a pull request will submit.
                                thanks all

                                vladisld 1 Reply Last reply Reply Quote 0
                                • vladisld
                                  vladisld @well cao last edited by

                                  @well-cao the decimal performance is lorder of magnitude (even two orders ) lower than float or integer. So if in critical path , it could really hurt the performance. Please take it into account.

                                  well cao 1 Reply Last reply Reply Quote 0
                                  • well cao
                                    well cao @vladisld last edited by

                                    @vladisld
                                    thanks for your information.
                                    you are absolutely right. decimal lack of performance
                                    do you have alternative scheme or any suggestion?

                                    well cao 1 Reply Last reply Reply Quote 0
                                    • well cao
                                      well cao @well cao last edited by

                                      @well-cao said in it should be a bug. size with decimal not works correctly:

                                      u have alternative scheme or any suggestion?

                                      look at this https://stackoverflow.com/questions/195116/is-there-a-faster-alternative-to-pythons-decimal
                                      and
                                      http://www.bytereef.org/mpdecimal/index.html

                                      the decimal is buit-in above python3.3+

                                      how do you think that?

                                      1 Reply Last reply Reply Quote 0
                                      • vladisld
                                        vladisld last edited by

                                        IMHO it is not good enough - the 'native' support for decimals that you've mentioned could be 'native' only for python. There is no native decimal support in hardware (at least not in common hardware that I know about) - so it still will be several times slower than float or int. I doubt anyone would agree to pay this price even if fractional order are used, right ? And especially so if fractional sizes are not used ( One should pay only for what is used and should not pay for what isn't).

                                        BTW it would be interesting to see how other platforms tackle this issue.

                                        well cao 1 Reply Last reply Reply Quote 1
                                        • well cao
                                          well cao @vladisld last edited by

                                          @vladisld @ab_trader
                                          thank you for the info.

                                          as @vladisld said ,it's not good enough. this approach can not sovle the float point issue fundamentally. cause the logic based on the size is very heavy used by the platform. but at least we have a workaround. better solution is welcome. life is short,let's move ahead. :)

                                          1 Reply Last reply Reply Quote 0
                                          • 1 / 1
                                          • First post
                                            Last post
                                          Copyright © 2016, 2017, 2018 NodeBB Forums | Contributors
                                          $(document).ready(function () { app.coldLoad(); }); }