Order Synchronization

I have added a issuee on github for this.

https://github.com/happydasch/btoandav20/issues/59

T H

@dasch if you know, please tell me.
Is there an ID that identifies a set of brackets instead of a tradeid?
Like the BatchID in the transaction log earlier.

T H

@dasch OMG! Thanks, I really appreciate it!!!!!

@T-H i don't understand what you mean with:

Is there an ID that identifies a set of brackets instead of a tradeid?

the BatchID is the id of the order that created the transaction. But this is an oanda thing. I don't use the BatchID in btonadav20.

T H

@dasch I see. Thanks!

T H

I tried several ways.
Unfortunately, it didn't improve.

Then I changed create_new_order() method in the following.

def create_new_order(self, price, exec_type):
    stopprice = price - (self.order_type_sign * self.max_range_price)
    limitprice = price + (self.order_type_sign * self.interval_price)

    self.tradeid_seq +=1
    
    orders = None
    if self.order_type == bt.Order.Buy:
        # orders = self.buy_bracket(size=self.order_size, price=price, exectype=exec_type, 
        #                     # tradeid=self.tradeid_seq, 
        #                     stopprice=stopprice, stopexec=bt.Order.Stop, 
        #                     limitprice=limitprice, limitexec=bt.Order.Limit)

        main_order = self.buy(size=self.order_size, price=price, exectype=exec_type, transmit=False)
        stop_order  = self.sell(price=stopprice, size=main_order.size, exectype=bt.Order.Stop,
                            transmit=False, parent=main_order)
        limit_order = self.sell(price=limitprice, size=main_order.size, exectype=bt.Order.Limit,
                            transmit=True, parent=main_order)
        
        orders = [main_order, stop_order, limit_order]
    elif self.order_type == bt.Order.Sell:
        # orders = self.sell_bracket(size=self.order_size, price=price, exectype=exec_type, 
        #                     # tradeid=self.tradeid_seq, 
        #                     stopprice=stopprice, stopexec=bt.Order.Stop, 
        #                     limitprice=limitprice, limitexec=bt.Order.Limit)

        main_order = self.sell(size=self.order_size, price=price, exectype=exec_type, transmit=False)
        stop_order  = self.buy(price=stopprice, size=main_order.size, exectype=bt.Order.Stop,
                            transmit=False, parent=main_order)
        limit_order = self.buy(price=limitprice, size=main_order.size, exectype=bt.Order.Limit,
                            transmit=True, parent=main_order)
        
        orders = [main_order, stop_order, limit_order]
    
    return orders

But, it still didn't improve.
There is cases where the order object differs from the actual state.

But then I realized that there is no id of transactions from 1845 to 1847.

Do you have any good solutions in regards to that?

NEW ORDER
( type:Sell, size:300, price:136.18999999999988, exectype:2(Limit), stopprice:146.18999999999988, limitprice:136.13999999999987 )

2020-09-11 19:36:37,706 [    INFO] *****, STORE NOTIF:, {'id': '1844', 'time': '1599820561.598982864', 'userID': 10733067, 'accountID': '101-009-10733067-005', 'batchID': '1844', 'requestID': '24724532290588330', 'type': 'LIMIT_ORDER', 'instrument': 'GBP_JPY', 'units': '-300.0', 'price': 136.19, 'timeInForce': 'GTC', 'positionFill': 'DEFAULT', 'triggerCondition': 'DEFAULT', 'reason': 'CLIENT_ORDER', 'clientExtensions': {'id': '1599755079.86117-565'}, 'takeProfitOnFill': {'price': 136.14, 'timeInForce': 'GTC', 'clientExtensions': {'id': '1599755079.86117-567'}}, 'stopLossOnFill': {'price': 146.19, 'timeInForce': 'GTC', 'clientExtensions': {'id': '1599755079.86117-566'}}}

2020-09-11 19:37:20,039 [    INFO] *****, STORE NOTIF:, Connection to v20 REST server at https://stream-fxpractice.oanda.com:443/v3/accounts/101-009-10733067-005/transactions/stream failed
2020-09-11 19:37:25,790 [    INFO] *****, STORE NOTIF:, Trying to reconnect streaming events (1 of -1)


2020-09-11 19:48:21,639 [    INFO] *****, STORE NOTIF:, {'id': '1848', 'time': '1599821264.782127550', 'userID': 10733067, 'accountID': '101-009-10733067-005', 'batchID': '1848', 'type': 'ORDER_FILL', 'orderID': '1846', 'clientOrderID': '1599755079.86117-567', 'instrument': 'GBP_JPY', 'units': '300.0', 'gainQuoteHomeConversionFactor': '1.0', 'lossQuoteHomeConversionFactor': '1.0', 'price': 136.14, 'fullVWAP': 136.14, 'fullPrice': {'type': 'PRICE', 'bids': [{'price': 136.126, 'liquidity': '250000'}], 'asks': [{'price': 136.14, 'liquidity': '250000'}], 'closeoutBid': 136.117, 'closeoutAsk': 136.149}, 'reason': 'TAKE_PROFIT_ORDER', 'pl': '15.0', 'financing': '0.0', 'commission': '0.0', 'guaranteedExecutionFee': '0.0', 'accountBalance': '3001745.978', 'tradesClosed': [{'tradeID': '1845', 'units': '300.0', 'price': 136.14, 'realizedPL': '15.0', 'financing': '0.0', 'guaranteedExecutionFee': '0.0', 'halfSpreadCost': '2.1'}], 'halfSpreadCost': '2.1'}
2020-09-11 19:48:21,640 [    INFO] *****, STORE NOTIF:, {'id': '1849', 'time': '1599821264.782127550', 'userID': 10733067, 'accountID': '101-009-10733067-005', 'batchID': '1848', 'type': 'ORDER_CANCEL', 'orderID': '1847', 'clientOrderID': '1599755079.86117-566', 'reason': 'LINKED_TRADE_CLOSED'}
2020-09-11 19:48:21,640 [    INFO] *****, STORE NOTIF:, Received external transaction ORDER_CANCEL with id 1849. Skipping transaction.

After this, the status of the previous new order(s) in the following.

Main Order: Accepted
TP Order: Submitted
SL Order: Submitted

Naturally, the main order have already been executed in the market.

i would need some code which reproduces this behaviour, else it is just guessing about the issue.

@T-H said in Order Synchronization:

But then I realized that there is no id of transactions from 1845 to 1847.

also for this, i am not sure why this happens. in btoandav20 the missing transactions should be fetched after a reconnection happened.

for msg_type, msg in response.parts():
                    # if a reconnection occurred
                    if reconnections > 0:
                        if last_id:
                            # get all transactions between the last seen and first from
                            # reconnected stream
                            old_transactions = self.get_transactions_since(
                                last_id)
                            for t in old_transactions:
                                if msg_type == "transaction.Transaction":
                                    if t.id > last_id:
                                        self._transaction(t.dict())
                                        last_id = t.id
                        reconnections = 0
                    if msg_type == "transaction.Transaction":
                        if not last_id or msg.id > last_id:
                            self._transaction(msg.dict())
                            last_id = msg.id

its this code block in the store.

T H

@dasch Oh... thank you for your information. Anyway, I'll check all of my code once.
By the way, In general, does this happen frequently with btoandav20? Or Is this rare in my case?

This issue was not yet posted, so I cannot tell if this happens often.

T H

@dasch I might have solved this problem.
Sometimes, I had created a new order and had canceled an other order in the one Next() iteration until now.
I tried to delay the timing of canceling order.
I have been running it all day, but now this problem isn't occurring.

Thank you for everything you've done for me!

Glad that the issue resolved.

T H

@dasch Hi. It's been a long time.
Please tell me your opinion.
I think the below code is not necessary.(That is "*** HERE ***" in code below)
Because if msg_type that got from a stream is "transaction.TransactionHeartbeat", this fetching code doesn't work.

oandav20store.py

                response = self.oapi_stream.transaction.stream(
                    self.p.account
                )
                # process response
                for msg_type, msg in response.parts():
                    # if a connection occurred
                    if reconnections > 0:
                        if reconnections > 0 and last_id:
                            # get all transactions between the last seen and first from
                            # reconnected stream
                            old_transactions = self.get_transactions_since(
                                last_id)
                            for t in old_transactions:
                                if msg_type == "transaction.Transaction":  # Is this code necessary? *** HERE ***
                                
                                    if t.id > last_id:
                                        self._transaction(t.dict())
                                        last_id = t.id
                        reconnections = 0
                    if msg_type == "transaction.Transaction":
                        if not last_id or msg.id > last_id:
                            self._transaction(msg.dict())
                            last_id = msg.id

What do you think about this?
(I'm sorry this isn't about backtrader.)

Thanks.

T H

only this "if".

if msg_type == "transaction.Transaction":  # Is this code necessary? *** HERE ***

the code is needed to ensure we handle a transaction

i ignore the transaction heartbeat there

Or asking differently, what do you intend to achieve by removing the line?

T H

@dasch Yes, That's right.
But this msg_type is current stream's msg_type.
What this code wants to handle is old transactions.

I think, I understand. Could you open an issue on GitHub with the msg_type in question.

T H

@dasch Thank you for understanding it.
But I’m not sure how to open an issue on Github.
Anyways, I tried to open this issue.
This is my first time.
Is this Okay?

https://github.com/happydasch/btoandav20/issues/60