it should be a bug. size with decimal not works correctly

well cao

i have confirm that is the size with decimal for example 3.5,0.01, 100.382 .etc
the notify_trade with the trade.justopened can not be fired sometimes.

look at the topics:
https://community.backtrader.com/topic/937/notify_trade-not-firing-on-subsequent-trades-when-using-tradeid
it also used the decimal size

you can reproduce the issue easily as the step:

:)set fixed size as any decimal number let's say 0.39
:)set and record the opened trade's unique id when trade.justopened fired
:)count the total trade ids after backtest finished using stop function
and you will see the number of trades you recorded with trade.justopened is not match the number Actually executed

well cao

https://community.backtrader.com/topic/1369/self-buy-size-1-5-does-not-work/7

another topic with decimal size.

since these issues have common point: float size

so it's not coincidence, i should be a bug. we should make bt work correctly with float size

ab_trader

there is no evidence of any bugs in the posts you referred to, just misunderstanding of the bt . and actually in the 2nd post it was an attempt to buy more than allowed by available cash.

if you think that the bug encountered, please post here the script, logs of the prices, orders and trades with the sizes.

well cao

@ab_trader
it think it's a bug
ok let code talk

import backtrader as bt
import shortuuid
from addict import Dict


class S1(bt.Strategy):
    params = (
        ('size', 1),
        ('signals_data', None),
        ('atr_times_for_break_even', 13),
        ('atr_times_for_stop_loss', 13),
        ('atr_times_for_take_profit', 26),
    )

    def __init__(self):
        self.data_index = 0
        self.signal_id = 0
        self.my_trades = Dict()
        self.my_trade_opened = Dict()
        self.my_trades_closed = Dict()
        self.not_recorded_by_opened = []
        if self.p.signals_data is None:
            raise Exception('signal_data_needed!')
            pass
        else:
            pass
        pass

    def next(self):
        s_signals_data = self.p.signals_data.iloc[self.data_index]
        atr_value = s_signals_data.atr
        if self.data_index > self.data.buflen() - 2:
            return
            pass
        if s_signals_data.trend_turn != 0:
            last_close_price = self.data.close[-1]
            # tradeid = shortuuid.uuid()
            tradeid = self.signal_id
            if s_signals_data.trend_turn == 1:
                stop_loss = last_close_price - atr_value * self.p.atr_times_for_stop_loss
                limit_price = last_close_price + atr_value * self.p.atr_times_for_take_profit
                orders_list = self.buy_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                               tradeid=tradeid, exectype=bt.Order.Market)
                self.signal_id += 1
                pass
            elif s_signals_data.trend_turn == -1:
                stop_loss = last_close_price + atr_value * self.p.atr_times_for_stop_loss
                limit_price = last_close_price - atr_value * self.p.atr_times_for_take_profit
                orders_list = self.sell_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                                tradeid=tradeid, exectype=bt.Order.Market)
                self.signal_id += 1
                pass
            pass
        self.data_index += 1

    def notify_trade(self, trade):
        if trade.justopened:
            tradeid = trade.tradeid
            self.my_trade_opened[tradeid] = tradeid
            self.my_trades[tradeid] = tradeid
        if trade.isclosed:
            tradeid = trade.tradeid
            self.my_trades_closed[tradeid] = tradeid
            try:
                del self.my_trades[trade.tradeid]
            except Exception:
                self.not_recorded_by_opened.append(tradeid)
                print('if there is no bug this should not appear', ' the tradeid ', tradeid,
                      ' not be recorded by trade.justopened when trade_isclosed fired')

    def stop(self):
        print('*' * 100)
        print('stop_executed')
        print('len_of_trade.justopened: ', len(self.my_trade_opened))
        print('len_of_not_recorded_by_trade.justopened: ', len(self.not_recorded_by_opened))
        print('view_of_the_dict_recorded_by_trade.justopened: ')
        print(self.my_trade_opened)
        pass

and the log:

if there is no bug this should not appear  the tradeid  17  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  18  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  16  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  22  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  25  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  27  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  26  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  24  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  28  not be recorded by trade.justopened when trade_isclosed fired
if there is no bug this should not appear  the tradeid  21  not be recorded by trade.justopened when trade_isclosed fired
****************************************************************************************************
stop_executed
len_of_trade.justopened:  39
len_of_not_recorded_by_trade.justopened:  10
view_of_the_dict_recorded_by_trade.justopened: 
{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 19: 19, 20: 20, 23: 23, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48}

well cao

@ab_trader
above comment is using a fractional size
if you give the set the size as an integer number everything works correctly!
the correct log is:

****************************************************************************************************
stop_executed
len_of_trade.justopened:  49
len_of_not_recorded_by_trade.justopened:  0
view_of_the_dict_recorded_by_trade.justopened: 
{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48}

notice this line

len_of_not_recorded_by_trade.justopened:  0

that means every trade fired by trade.isclosed is record by trade.justopened

well cao

@ab_trader
feel free to comment if i havn't make it clearly

ab_trader

i have nothing to comment here since (1) there is not enough information logged, (2) i can't run this script by myself without significant modification and (3) script is quite complex which may induce additional errors not related to the behavior we are trying to understand. For example, tradeid is internal bt counter and you are passing some unique tradeids with the orders. therefore you may interfere with that internal mechanism.

well cao

@ab_trader
thanks for your reply, but from your comment , i don't think you are the owner of this great backtest framework or even a programmer. excuse me if you get offended :)
because if you are the owner or contributor to this framework you should know the framework's source code or mechanism.
:) the tradeid. check this blog--https://www.backtrader.com/blog/posts/2015-10-05-multitrades/multitrades/
the unique tradeid for every trade is for the sake of multiple trades. that's why unique tradeid comes from
i don't think this "bug" has a bearing on tradeid. even so it's still a bug.
:) after some dig into the source code. i found that some logic based on the size.
as we know in computer science if the size is integer for example

a=2
b=2
print(a==2)
# the result will be True
#but if we use fractional size for example 
a=2.0
b=2.00000000000000035
print(a==b)
#the result is False.
#but may sometime our "a" variable probaly become the value of "b"

just a hint. maybe i'm wrong if so let me know . thanks

if you think it's neccessary to i will post my full code version . thank you

ab_trader

@well-cao said in it should be a bug. size with decimal not works correctly:

just a hint. maybe i'm wrong if so let me know . thanks
if you think it's neccessary to i will post my full code version . thank you

if you have any evidence confirming that the behavior observed is a bug, than you are welcome to share corresponding information to community, and maybe even make pull request fixing this bug to https://github.com/backtrader2/backtrader (community version bt).

vladisld

hey @well-cao, could you please provide a full code so we could play with it around ? I'm not sure how to repro using the strategy code above ( for example: what need to be passed as a strategy parameter signals_data ? ). It will be event better if you could use the standard test data sets provided with the backtrader itself - this way it will be easier for us to repro.

well cao

@vladisld
thanks for your reply
here is the full version code, the data read from csv file named "test_data.csv"
i will attach it with the code:

import backtrader as bt
import pandas as pd
from addict import Dict  # pip install addict


class firstStrategy(bt.Strategy):
    params = (
        ('size', 1),
        ('atr_times_for_break_even', 13),
        ('atr_times_for_stop_loss', 13),
        ('atr_times_for_take_profit', 26),
    )

    def __init__(self):
        self.bars_index = 0
        self.signal_id = 0
        self.my_trades = Dict()
        self.my_trade_opened = Dict()
        self.my_trades_closed = Dict()
        self.not_recorded_by_opened = []

    def next(self):
        if self.data.trend_turn[0] != 0:
            atr_value = self.data.atr[0]
            last_close_price = self.data.close[0]
            tradeid = self.signal_id
            # tradeid=shortuuid.uuid()
            if self.data.trend_turn[0] == 1:
                # stop_loss = last_close_price - atr_value * self.p.atr_times_for_stop_loss
                stop_loss = last_close_price - atr_value * 13
                # limit_price = last_close_price + atr_value * self.p.atr_times_for_take_profit
                limit_price = last_close_price + atr_value * 13
                orders_list = self.buy_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                               tradeid=tradeid, exectype=bt.Order.Market)
                # main_side = self.buy(size=self.p.size, exectype=bt.Order.Market, tradeid=tradeid, transmit=False)
                # self.sell(size=self.p.size, exectype=bt.Order.Stop, tradeid=tradeid, price=stop_loss, parent=main_side,
                #           transmit=False)
                # self.sell(size=self.p.size, exectype=bt.Order.Limit, price=limit_price, tradeid=tradeid,
                #           parent=main_side, transmit=True)

                pass
            else:
                # stop_loss = last_close_price + atr_value * self.p.atr_times_for_stop_loss
                stop_loss = last_close_price + atr_value * 13
                # limit_price = last_close_price - atr_value * self.p.atr_times_for_take_profit
                limit_price = last_close_price - atr_value * 13
                orders_list = self.sell_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size,
                                                tradeid=tradeid, exectype=bt.Order.Market, price=last_close_price)
                # main_side = self.sell(size=self.p.size, exectype=bt.Order.Market, tradeid=tradeid, transmit=False)
                # self.buy(size=self.p.size, exectype=bt.Order.Stop, tradeid=tradeid, price=stop_loss, parent=main_side,
                #          transmit=False)
                # self.buy(size=self.p.size, exectype=bt.Order.Limit, price=limit_price, tradeid=tradeid,
                #          parent=main_side, transmit=True)

            self.signal_id += 1
        pass

    def notify_trade(self, trade):
        if trade.justopened:
            tradeid = trade.tradeid
            # print('trade_size:', trade.size)
            self.my_trade_opened[tradeid] = tradeid
            self.my_trades[tradeid] = tradeid
        if trade.isclosed:
            tradeid = trade.tradeid
            # print('closed:----tradeid:', tradeid)
            self.my_trades_closed[tradeid] = tradeid
            try:
                del self.my_trades[trade.tradeid]
            except Exception:
                self.not_recorded_by_opened.append(tradeid)
                print('if there is no bug this should not appear', ' the tradeid ', tradeid,
                      ' not be recorded by trade.justopened when trade_isclosed fired')

    def stop(self):
        print('*' * 100)
        print('stop_executed')
        print('len_of_trade.justopened: ', len(self.my_trade_opened))
        print('len_of_not_recorded_by_trade.justopened: ', len(self.not_recorded_by_opened))
        print('view_of_the_dict_recorded_by_trade.justopened: ')
        print(self.my_trade_opened)
        pass


class MyPandasData(bt.feeds.PandasData):
    # 添加额外的数据列--extra customized columns
    lines = (
        'trend_turn',
        'atr',
    )
    params = (
        ('datetime', 0),
        ('open', -1),
        ('high', -1),
        ('low', -1),
        ('close', -1),
        ('volume', 5),
        ('openinterest', None),
        ('trend_turn', 8),
        ('atr', 9),
    )
    pass


signals_data = pd.read_csv('test_data.csv')
signals_data['time'] = pd.to_datetime(signals_data['time'])
data = MyPandasData(
    dataname=signals_data,
    timeframe=bt.TimeFrame.Minutes,
)

cerebro = bt.Cerebro()
# some size for test below:
# fractional size:
# size = 0.11687
# size = 3.2646498
size = 1.3983
# integer size
# size = 4
# size = 3.0  #
# size = 3
# size = 8
cerebro.addstrategy(firstStrategy, size=size)
cerebro.adddata(data)
startcash = 100000
cerebro.broker.setcash(startcash)
my_strategy = cerebro.run()[0]
# Finally plot the end results
# cerebro.plot(style='candlestick')

the csv data file----csv file uploaed forbidden. you can download the csv file from here:

https://drive.google.com/file/d/1o5M-KaJeQuP9WK8INar3ZUinipVEGHwd/view?usp=sharing

vladisld

It seems that there is indeed a problem with fractional order sizes and trades. I believe the same problem was already briefly mentioned in the following post: https://community.backtrader.com/topic/2840/race-condition-bug-w-orders-and-trades/18

Just as a side note:

The backtrader author mentioned the support for the fractional order sizes for crypto-currencies trading: https://www.backtrader.com/blog/posts/2019-08-29-fractional-sizes/fractional-sizes/#trading-cryptocurrencies-fractions

The support for added in CommissionInfo (getsize method) to return the size of the order given the price and the target percentage. It was intended to be used in order_target_value strategy method as well as to be used by sizers.

Back to the problem raised by @well-cao:

Debugging an issue a little bit it seems that accumulating a position using fractional orders will hit the floating point representation limits causing the small errors to sneak into the position size. This will in turn cause the position not to be closed properly (leaving the tiny amount of position). Since the trade is only declared closed once the position size is absolute zero - the trade will always stay opened.

Simple example (unrelated to backtrader - just python console - here the f is representing the position size):

>>> f = 0.0
>>> f += 1.3983
>>> f
1.3983
>>> f += 1.3983
>>> f
2.7966
>>> f += 1.3983
>>> f
4.1949000000000005
>>> f -= 1.3983
>>> f
2.7966000000000006
>>> f -= 1.3983
>>> f
1.3983000000000005
>>> f -= 1.3983
>>> f
4.440892098500626e-16

I'm not sure what the right fix should be though and will appreciate any further ideas. ( I can share my debug session and data if needed )

well cao

@vladisld @ab_trader
thanks for your kind reply.
i have noticed that article before - https://www.backtrader.com/blog/posts/2019-08-29-fractional-sizes/fractional-sizes/#trading-cryptocurrencies-fractions
if not i will think it's my fault because as the document describe the size should be integer

 Args:
            order: the order object which has (completely or partially)
                generated this update
            size (int): amount to update the order
                if size has the same sign as the current trade a

because of that article mentioned above so i think it maybe a bug with fractional size since we allow fractional size to use.

that is nice hint for the f example. that 's where i give my doubt where the problem comes from. i had try to fix it but with no luck.
any opinion , pull request,or patch to work around is appriceated. thanks

well cao

@vladisld @ab_trader
after some debug.
i got it work by modify some source code.
the file is "/backtrader/position.py"
first import decimal module

from decimal import Decimal #add this line first

and then

modify the update function---near line number: 165
replace this line self.size += size with
self.size = float(Decimal(str(self.size))+Decimal(str(size)))

and so far it works as expected. if i'm right maybe a pull request will submit.
thanks all

vladisld

@well-cao the decimal performance is lorder of magnitude (even two orders ) lower than float or integer. So if in critical path , it could really hurt the performance. Please take it into account.

well cao

@vladisld
thanks for your information.
you are absolutely right. decimal lack of performance
do you have alternative scheme or any suggestion?

well cao

@well-cao said in it should be a bug. size with decimal not works correctly:

u have alternative scheme or any suggestion?

look at this https://stackoverflow.com/questions/195116/is-there-a-faster-alternative-to-pythons-decimal
and
http://www.bytereef.org/mpdecimal/index.html

the decimal is buit-in above python3.3+

how do you think that?

vladisld

IMHO it is not good enough - the 'native' support for decimals that you've mentioned could be 'native' only for python. There is no native decimal support in hardware (at least not in common hardware that I know about) - so it still will be several times slower than float or int. I doubt anyone would agree to pay this price even if fractional order are used, right ? And especially so if fractional sizes are not used ( One should pay only for what is used and should not pay for what isn't).

BTW it would be interesting to see how other platforms tackle this issue.

well cao

@vladisld @ab_trader
thank you for the info.

as @vladisld said ,it's not good enough. this approach can not sovle the float point issue fundamentally. cause the logic based on the size is very heavy used by the platform. but at least we have a workaround. better solution is welcome. life is short,let's move ahead. :)