incorrectly calculated order size in order_target_value

Ed Bartosh

When I close open position using order_target_value and order_target percent(which is using order_target_value) the size is incorrectly calculated. order_target_size works just fine:

Here is the log output for both cases:

Size -653 is incorrectly calculated when using self.order_target_percent(self.data0, 0). It should be -654.

info: 2015-01-05 11:20:00, short entry order 1, stock = AAPL, size = 654
info: 2015-01-05 11:21:00, ORDER 1 EXECUTED, Price: 106.68, Cost: -69768.72, Comm 0.00, Cost basis: 106.68
info: 2015-01-05 11:22:00, (short position) stop loss order 2, stock = AAPL, entry price = 106.68, stop price = 107.826236937
info: 2015-01-05 15:30:00, eod canceling: AAPL_min: order = 2, amount = 654
info: 2015-01-05 15:30:00, eod exit position: AAPL: order = 3, amount = -653

It woks ok if I use self.order_target_size(self.data0, 0):

Starting Portfolio Value: 10000000.00
info: 2015-01-05 11:20:00, short entry order 1, stock = AAPL, size = 654
info: 2015-01-05 11:21:00, ORDER 1 EXECUTED, Price: 106.68, Cost: -69768.72, Comm 0.00, Cost basis: 106.68
info: 2015-01-05 11:22:00, (short position) stop loss order 2, stock = AAPL, entry price = 106.68, stop price = 107.826236937
info: 2015-01-05 15:30:00, eod canceling: AAPL_min: order = 2, amount = 654
info: 2015-01-05 15:30:00, eod exit position: AAPL: order = 3, amount = 654

I'm guessing the reason is incorrect rounding somewhere in order_target* code.

backtrader

The 0% or 0 value cases need probably be handled separately. If the division returned 653 rather than 654 is because the result of calculating the exit size as a function of the current value and price yielded something below 654 and above 653

The puzzling thing here is: -653, with a negative sign?

Ed Bartosh

The puzzling thing here is: -653, with a negative sign?

Looks like a bug in order_target_value code to me:

if target > value:
            size = comminfo.getsize(price, target - value)
            if possize >= 0:
                return self.buy(data=data, size=size,
                                price=price, plimit=plimit,
                                exectype=exectype, valid=valid,
                                tradeid=tradeid, **kwargs)
            else:
                return self.sell(data=data, size=size,
                                 price=price, plimit=plimit,
                                 exectype=exectype, valid=valid,
                                 tradeid=tradeid, **kwargs)

In this particular case value is -69768.72, target is 0 and size is 653. So good so far. However, as position size is negative(-654) we end up calling self.sell instead of self.buy.

Hope it helps.

backtrader

sell, buy and close use abs to turn any negative value into a positive one to exactly avoid those use cases.

The real question is: how have you ended up with a negative value (i.e.: -69768.72?)

Ed Bartosh

@backtrader This is what self.broker.getvalue(datas=[data]) returns for short position I guess. This is a first trade, btw. So, the only order that has been executed is a short sell of 654 shares of AAPL.

backtrader

The negative value of the asset doesn't say how you end with -653 (with negative sign), because the operations use abs. At the end of the day even if the internal calculation yielded -653, it should have the same sign (positive) application via order_target_size as soon as it gets through buy, sell or close. And this happens already internally in order_target_xxx.

May it be that you are logging from inside those methods with a modified version?

Ed Bartosh

@backtrader no, I didn't modify any backtester code.

hm, interesting. When I print return value of _get_value in bbroker.py I get positive number. However, in strategy.py, immediately after the call it's negative. Magic :) !

Here is the diff:

diff --git a/backtrader/brokers/bbroker.py b/backtrader/brokers/bbroker.py
index 949337a..aa0fb0b 100644
--- a/backtrader/brokers/bbroker.py
+++ b/backtrader/brokers/bbroker.py
@@ -384,6 +384,8 @@ class BackBroker(bt.BrokerBase):
         self._leverage = pos_value / (pos_value_unlever or 1.0)
         self._unrealized = unrealized
 
+        print("broker:", self._value if not lever else self._valuelever)
+
         return self._value if not lever else self._valuelever
 
     def get_leverage(self):
diff --git a/backtrader/strategy.py b/backtrader/strategy.py
index ecd5cc4..4c47e6f 100644
--- a/backtrader/strategy.py
+++ b/backtrader/strategy.py
@@ -800,6 +800,9 @@ class Strategy(with_metaclass(MetaStrategy, StrategyBase)):
             data = self.data
 
         possize = self.getposition(data, self.broker).size
+
+        print("strategy:", self.broker.getvalue(datas=[data]))
+
         value = self.broker.getvalue(datas=[data])
         comminfo = self.broker.getcommissioninfo(data)

And the output:

broker: 100320.46
broker: 100340.08
broker: 100372.78
broker: 100412.02
info: 2015-01-05 15:30:00, eod canceling: min: order = 2, amount = 654
strategy: -69356.7
info: 2015-01-05 15:30:00, eod exit position: AAPL: order = 3, amount = -653

I'm on this commit:

commit d4e56d6042afcbe363e3b765e87991e81d4b11c9
Author: mementum <mementum@users.noreply.github.com>
Date:   Mon Jan 16 23:09:30 2017 +0100

    Remove iscfd leftover from assimilation to iscash

backtrader

It's still 100% unclear where that

info: ... -653

comes from.

And it's no magic. In one case the overall portfolio value is being requested, which includes the cash in the system. In the 2nd case the valuation of a single asset is being requested and the short assets are returned as negative.

Ed Bartosh

The real question is: how have you ended up with a negative value (i.e.: -69768.72?)
...
In the 2nd case the valuation of a single asset is being requested and the short assets are returned as negative.

I guess you've answered your question, right?

It's still 100% unclear ...

If you sell 653 shares, order.size will be -653. What's still unclear here?

To close short position we should buy, but we're selling. It looks like a bug to me.

backtrader

You said it was magic and the magic was explained.

And finally you said where the -653 comes from. From order.size. Which reveals what is happening. Logs only help so much if they don't give any hint as to where they come from.

There was a change in the semantics of get_value with the addition of the parameter shortcash to the broker, to let short positions either purely add cash to the system or detract it after the calculation was made.

With that in mind, the reported value of a short position is negative and changing the value of the parameter, it turns positive. This new semantics are not taken into account in order_target_value and hence the wrong sell instead of buy

backtrader

Tryng to comprehend the complete use case.

How did you open the short position in the 1st place? Directly with sell, with order_target_size?

Ed Bartosh

@backtrader with sell

backtrader

@Ed-Bartosh Notwithstanding the needed look into order_target_value to better manage the new semantics explained above.

Why wouldn't you close the position with close? (which is a member of the buy, sell family) which already takes into account the exact position and acts accordingly.

Ed Bartosh

@backtrader Why wouldn't you close the position with close?

already did that. I used order_target as my zipline code that I'm porting to backtrader used order_target* APIs.