it should be a bug. size with decimal not works correctly
-
i have confirm that is the size with decimal for example 3.5,0.01, 100.382 .etc
the notify_trade with the trade.justopened can not be fired sometimes.look at the topics:
https://community.backtrader.com/topic/937/notify_trade-not-firing-on-subsequent-trades-when-using-tradeid
it also used the decimal sizeyou can reproduce the issue easily as the step:
:)set fixed size as any decimal number let's say 0.39
:)set and record the opened trade's unique id when trade.justopened fired
:)count the total trade ids after backtest finished using stop function
and you will see the number of trades you recorded with trade.justopened is not match the number Actually executed -
https://community.backtrader.com/topic/1369/self-buy-size-1-5-does-not-work/7
another topic with decimal size.
since these issues have common point: float size
so it's not coincidence, i should be a bug. we should make bt work correctly with float size
-
there is no evidence of any bugs in the posts you referred to, just misunderstanding of the
bt
. and actually in the 2nd post it was an attempt to buy more than allowed by available cash.if you think that the bug encountered, please post here the script, logs of the prices, orders and trades with the sizes.
-
@ab_trader
it think it's a bug
ok let code talkimport backtrader as bt import shortuuid from addict import Dict class S1(bt.Strategy): params = ( ('size', 1), ('signals_data', None), ('atr_times_for_break_even', 13), ('atr_times_for_stop_loss', 13), ('atr_times_for_take_profit', 26), ) def __init__(self): self.data_index = 0 self.signal_id = 0 self.my_trades = Dict() self.my_trade_opened = Dict() self.my_trades_closed = Dict() self.not_recorded_by_opened = [] if self.p.signals_data is None: raise Exception('signal_data_needed!') pass else: pass pass def next(self): s_signals_data = self.p.signals_data.iloc[self.data_index] atr_value = s_signals_data.atr if self.data_index > self.data.buflen() - 2: return pass if s_signals_data.trend_turn != 0: last_close_price = self.data.close[-1] # tradeid = shortuuid.uuid() tradeid = self.signal_id if s_signals_data.trend_turn == 1: stop_loss = last_close_price - atr_value * self.p.atr_times_for_stop_loss limit_price = last_close_price + atr_value * self.p.atr_times_for_take_profit orders_list = self.buy_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size, tradeid=tradeid, exectype=bt.Order.Market) self.signal_id += 1 pass elif s_signals_data.trend_turn == -1: stop_loss = last_close_price + atr_value * self.p.atr_times_for_stop_loss limit_price = last_close_price - atr_value * self.p.atr_times_for_take_profit orders_list = self.sell_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size, tradeid=tradeid, exectype=bt.Order.Market) self.signal_id += 1 pass pass self.data_index += 1 def notify_trade(self, trade): if trade.justopened: tradeid = trade.tradeid self.my_trade_opened[tradeid] = tradeid self.my_trades[tradeid] = tradeid if trade.isclosed: tradeid = trade.tradeid self.my_trades_closed[tradeid] = tradeid try: del self.my_trades[trade.tradeid] except Exception: self.not_recorded_by_opened.append(tradeid) print('if there is no bug this should not appear', ' the tradeid ', tradeid, ' not be recorded by trade.justopened when trade_isclosed fired') def stop(self): print('*' * 100) print('stop_executed') print('len_of_trade.justopened: ', len(self.my_trade_opened)) print('len_of_not_recorded_by_trade.justopened: ', len(self.not_recorded_by_opened)) print('view_of_the_dict_recorded_by_trade.justopened: ') print(self.my_trade_opened) pass
and the log:
if there is no bug this should not appear the tradeid 17 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 18 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 16 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 22 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 25 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 27 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 26 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 24 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 28 not be recorded by trade.justopened when trade_isclosed fired if there is no bug this should not appear the tradeid 21 not be recorded by trade.justopened when trade_isclosed fired **************************************************************************************************** stop_executed len_of_trade.justopened: 39 len_of_not_recorded_by_trade.justopened: 10 view_of_the_dict_recorded_by_trade.justopened: {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 19: 19, 20: 20, 23: 23, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48}
-
@ab_trader
above comment is using a fractional size
if you give the set the size as an integer number everything works correctly!
the correct log is:**************************************************************************************************** stop_executed len_of_trade.justopened: 49 len_of_not_recorded_by_trade.justopened: 0 view_of_the_dict_recorded_by_trade.justopened: {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48}
notice this line
len_of_not_recorded_by_trade.justopened: 0
that means every trade fired by trade.isclosed is record by trade.justopened
-
@ab_trader
feel free to comment if i havn't make it clearly -
i have nothing to comment here since (1) there is not enough information logged, (2) i can't run this script by myself without significant modification and (3) script is quite complex which may induce additional errors not related to the behavior we are trying to understand. For example,
tradeid
is internalbt
counter and you are passing some uniquetradeid
s with the orders. therefore you may interfere with that internal mechanism. -
@ab_trader
thanks for your reply, but from your comment , i don't think you are the owner of this great backtest framework or even a programmer. excuse me if you get offended :)
because if you are the owner or contributor to this framework you should know the framework's source code or mechanism.
:) the tradeid. check this blog--https://www.backtrader.com/blog/posts/2015-10-05-multitrades/multitrades/
the unique tradeid for every trade is for the sake of multiple trades. that's why unique tradeid comes from
i don't think this "bug" has a bearing on tradeid. even so it's still a bug.
:) after some dig into the source code. i found that some logic based on the size.
as we know in computer science if the size is integer for examplea=2 b=2 print(a==2) # the result will be True #but if we use fractional size for example a=2.0 b=2.00000000000000035 print(a==b) #the result is False. #but may sometime our "a" variable probaly become the value of "b"
just a hint. maybe i'm wrong if so let me know . thanks
if you think it's neccessary to i will post my full code version . thank you
-
@well-cao said in it should be a bug. size with decimal not works correctly:
just a hint. maybe i'm wrong if so let me know . thanks
if you think it's neccessary to i will post my full code version . thank youif you have any evidence confirming that the behavior observed is a bug, than you are welcome to share corresponding information to community, and maybe even make pull request fixing this bug to https://github.com/backtrader2/backtrader (community version
bt
). -
hey @well-cao, could you please provide a full code so we could play with it around ? I'm not sure how to repro using the strategy code above ( for example: what need to be passed as a strategy parameter
signals_data
? ). It will be event better if you could use the standard test data sets provided with the backtrader itself - this way it will be easier for us to repro. -
@vladisld
thanks for your reply
here is the full version code, the data read from csv file named "test_data.csv"
i will attach it with the code:import backtrader as bt import pandas as pd from addict import Dict # pip install addict class firstStrategy(bt.Strategy): params = ( ('size', 1), ('atr_times_for_break_even', 13), ('atr_times_for_stop_loss', 13), ('atr_times_for_take_profit', 26), ) def __init__(self): self.bars_index = 0 self.signal_id = 0 self.my_trades = Dict() self.my_trade_opened = Dict() self.my_trades_closed = Dict() self.not_recorded_by_opened = [] def next(self): if self.data.trend_turn[0] != 0: atr_value = self.data.atr[0] last_close_price = self.data.close[0] tradeid = self.signal_id # tradeid=shortuuid.uuid() if self.data.trend_turn[0] == 1: # stop_loss = last_close_price - atr_value * self.p.atr_times_for_stop_loss stop_loss = last_close_price - atr_value * 13 # limit_price = last_close_price + atr_value * self.p.atr_times_for_take_profit limit_price = last_close_price + atr_value * 13 orders_list = self.buy_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size, tradeid=tradeid, exectype=bt.Order.Market) # main_side = self.buy(size=self.p.size, exectype=bt.Order.Market, tradeid=tradeid, transmit=False) # self.sell(size=self.p.size, exectype=bt.Order.Stop, tradeid=tradeid, price=stop_loss, parent=main_side, # transmit=False) # self.sell(size=self.p.size, exectype=bt.Order.Limit, price=limit_price, tradeid=tradeid, # parent=main_side, transmit=True) pass else: # stop_loss = last_close_price + atr_value * self.p.atr_times_for_stop_loss stop_loss = last_close_price + atr_value * 13 # limit_price = last_close_price - atr_value * self.p.atr_times_for_take_profit limit_price = last_close_price - atr_value * 13 orders_list = self.sell_bracket(stopprice=stop_loss, limitprice=limit_price, size=self.p.size, tradeid=tradeid, exectype=bt.Order.Market, price=last_close_price) # main_side = self.sell(size=self.p.size, exectype=bt.Order.Market, tradeid=tradeid, transmit=False) # self.buy(size=self.p.size, exectype=bt.Order.Stop, tradeid=tradeid, price=stop_loss, parent=main_side, # transmit=False) # self.buy(size=self.p.size, exectype=bt.Order.Limit, price=limit_price, tradeid=tradeid, # parent=main_side, transmit=True) self.signal_id += 1 pass def notify_trade(self, trade): if trade.justopened: tradeid = trade.tradeid # print('trade_size:', trade.size) self.my_trade_opened[tradeid] = tradeid self.my_trades[tradeid] = tradeid if trade.isclosed: tradeid = trade.tradeid # print('closed:----tradeid:', tradeid) self.my_trades_closed[tradeid] = tradeid try: del self.my_trades[trade.tradeid] except Exception: self.not_recorded_by_opened.append(tradeid) print('if there is no bug this should not appear', ' the tradeid ', tradeid, ' not be recorded by trade.justopened when trade_isclosed fired') def stop(self): print('*' * 100) print('stop_executed') print('len_of_trade.justopened: ', len(self.my_trade_opened)) print('len_of_not_recorded_by_trade.justopened: ', len(self.not_recorded_by_opened)) print('view_of_the_dict_recorded_by_trade.justopened: ') print(self.my_trade_opened) pass class MyPandasData(bt.feeds.PandasData): # 添加额外的数据列--extra customized columns lines = ( 'trend_turn', 'atr', ) params = ( ('datetime', 0), ('open', -1), ('high', -1), ('low', -1), ('close', -1), ('volume', 5), ('openinterest', None), ('trend_turn', 8), ('atr', 9), ) pass signals_data = pd.read_csv('test_data.csv') signals_data['time'] = pd.to_datetime(signals_data['time']) data = MyPandasData( dataname=signals_data, timeframe=bt.TimeFrame.Minutes, ) cerebro = bt.Cerebro() # some size for test below: # fractional size: # size = 0.11687 # size = 3.2646498 size = 1.3983 # integer size # size = 4 # size = 3.0 # # size = 3 # size = 8 cerebro.addstrategy(firstStrategy, size=size) cerebro.adddata(data) startcash = 100000 cerebro.broker.setcash(startcash) my_strategy = cerebro.run()[0] # Finally plot the end results # cerebro.plot(style='candlestick')
the csv data file----csv file uploaed forbidden. you can download the csv file from here:
https://drive.google.com/file/d/1o5M-KaJeQuP9WK8INar3ZUinipVEGHwd/view?usp=sharing
-
It seems that there is indeed a problem with fractional order sizes and trades. I believe the same problem was already briefly mentioned in the following post: https://community.backtrader.com/topic/2840/race-condition-bug-w-orders-and-trades/18
Just as a side note:
The backtrader author mentioned the support for the fractional order sizes for crypto-currencies trading: https://www.backtrader.com/blog/posts/2019-08-29-fractional-sizes/fractional-sizes/#trading-cryptocurrencies-fractions
The support for added in CommissionInfo (getsize method) to return the size of the order given the price and the target percentage. It was intended to be used in
order_target_value
strategy method as well as to be used by sizers.Back to the problem raised by @well-cao:
Debugging an issue a little bit it seems that accumulating a position using fractional orders will hit the floating point representation limits causing the small errors to sneak into the position size. This will in turn cause the position not to be closed properly (leaving the tiny amount of position). Since the trade is only declared closed once the position size is absolute zero - the trade will always stay opened.
Simple example (unrelated to backtrader - just python console - here the
f
is representing the position size):>>> f = 0.0 >>> f += 1.3983 >>> f 1.3983 >>> f += 1.3983 >>> f 2.7966 >>> f += 1.3983 >>> f 4.1949000000000005 >>> f -= 1.3983 >>> f 2.7966000000000006 >>> f -= 1.3983 >>> f 1.3983000000000005 >>> f -= 1.3983 >>> f 4.440892098500626e-16
I'm not sure what the right fix should be though and will appreciate any further ideas. ( I can share my debug session and data if needed )
-
@vladisld @ab_trader
thanks for your kind reply.
i have noticed that article before - https://www.backtrader.com/blog/posts/2019-08-29-fractional-sizes/fractional-sizes/#trading-cryptocurrencies-fractions
if not i will think it's my fault because as the document describe the size should be integerArgs: order: the order object which has (completely or partially) generated this update size (int): amount to update the order if size has the same sign as the current trade a
because of that article mentioned above so i think it maybe a bug with fractional size since we allow fractional size to use.
that is nice hint for the
f
example. that 's where i give my doubt where the problem comes from. i had try to fix it but with no luck.
any opinion , pull request,or patch to work around is appriceated. thanks -
@vladisld @ab_trader
after some debug.
i got it work by modify some source code.
the file is "/backtrader/position.py"
first import decimal modulefrom decimal import Decimal #add this line first
and then
modify the
update
function---near line number: 165
replace this lineself.size += size
with
self.size = float(Decimal(str(self.size))+Decimal(str(size)))
and so far it works as expected. if i'm right maybe a pull request will submit.
thanks all -
@well-cao the decimal performance is lorder of magnitude (even two orders ) lower than float or integer. So if in critical path , it could really hurt the performance. Please take it into account.
-
@vladisld
thanks for your information.
you are absolutely right. decimal lack of performance
do you have alternative scheme or any suggestion? -
@well-cao said in it should be a bug. size with decimal not works correctly:
u have alternative scheme or any suggestion?
look at this https://stackoverflow.com/questions/195116/is-there-a-faster-alternative-to-pythons-decimal
and
http://www.bytereef.org/mpdecimal/index.htmlthe decimal is buit-in above python3.3+
how do you think that?
-
IMHO it is not good enough - the 'native' support for decimals that you've mentioned could be 'native' only for python. There is no native decimal support in hardware (at least not in common hardware that I know about) - so it still will be several times slower than float or int. I doubt anyone would agree to pay this price even if fractional order are used, right ? And especially so if fractional sizes are not used ( One should pay only for what is used and should not pay for what isn't).
BTW it would be interesting to see how other platforms tackle this issue.
-
@vladisld @ab_trader
thank you for the info.as @vladisld said ,it's not good enough. this approach can not sovle the float point issue fundamentally. cause the logic based on the size is very heavy used by the platform. but at least we have a workaround. better solution is welcome. life is short,let's move ahead. :)