Why skip order.Margin status?

Taewoo Kim

In the quickstart tutorial, what is the logic behind why you skip margin orders?

    elif order.status in [order.Canceled, order.Margin, order.Rejected]:
        self.log('Order Canceled/Margin/Rejected')

I can understand why cancelled / rejected orders should set self.order to none, but not why margin order.

According to the docs:

Order.Margin: the order execution would imply a margin call and the previously accepted order has been taken off the system

However, I'm placing a tiny orders (account $100k value, value of position < $1k).

Is there a quick way to debug this?

backtrader

It seems you misunderstand what's being done with self.order = None (which is mentioned but not shown): self.order is a sentinel in the strategy to avoid order accumulation, i.e.: an order will not be sent to the broker if another order has not yet been fully processed (Completed, Rejected, ...)

@Taewoo-Kim said in Why skip order.Margin status?:

Order.Margin: the order execution would imply a margin call and the previously accepted order has been taken off the system

As you may see in the documentation you quote: the order is not being skipped. The order is no longer in the system and the strategy sets the sentinel to None in the example to allow new orders to be produced and sent.

@Taewoo-Kim said in Why skip order.Margin status?:

However, I'm placing a tiny orders (account $100k value, value of position < $1k).
Is there a quick way to debug this?

This seems to imply that you are getting Order.Margin, but without some actual code it is impossible to say what may be wrong. It could be as simple as having 100 orders in the broker each for around $1k ... order 101 wouldn't have cash enough.

Taewoo Kim

This seems to imply that you are getting Order.Margin, but without some actual code it is impossible to say what may be wrong. It could be as simple as having 100 orders in the broker each for around $1k ... order 101 wouldn't have cash enough.

Ok, so I went back and noticed that most buy orders (not closing short positions, but conversion of cash to long position) were ending up w/o order.Margin status in notify_order.

I am following Oanda's unit size calculator to limit my position size to under 1% of the account balance.

Assuming 100k account and 10:1 leverage, this yields in 8900'ish unit size:

alt text

My strategy calculates this correctly :

[INFO] 23:39:42 {'volume': 722.0, 'datetime': datetime.datetime(2017, 5, 29, 0, 0), 'close': 1.1166399999999999}
[INFO] 23:39:42 [NEXT 1] Signals: Buy True      Sell False
Order size 8948.505823240164

However, even after setting 100k

cerebro.broker.setcash(cash)

I still keep getting order.Margin.

Any ideas?

Taewoo Kim

strategy code is simple..

class TemplateSt(bt.Strategy):
	params = (
		( 'timeframe_0_tf', '1Min'),
	)

	def __init__(self):
		self.sma = bt.ind.MovingAverageSimple(period=5)
		self.buysig = bt.ind.CrossDown(self.datas[0].close[0], self.sma)
		self.sellsig = bt.ind.CrossUp(self.datas[0].close[0], self.sma)
		

	def next(self):
		
		if(self.buysig is None or self.sellsig is None):
			return


		order_size = get_units()

		if(self.buysig):
			# print("Buying")
			if(self.position.size <0):
				
				if(self.last_bar_ordered != 0 and  not_much_price_diff is False):
					logging.info("[NEXT {}] Closing short position".format(len(self)))
					self.close()
				else:
					logging.info("[NEXT {}] Price diff small to close short position. Skipping".format(len(self)))
					
			elif(self.position.size==0):

				logging.info("[NEXT {}] Creating buy order".format(len(self)))
				logging.info("[NEXT {}] Order Size: {}".format(len(self), order_size))
		
				self.order = self.buy(size=order_size)

		elif (self.sellsig):
			# print("Selling")
			if(self.position.size > 0):
				if(self.last_bar_ordered != 0 and not_much_price_diff is False):
					logging.info("[NEXT {}] Closing long position".format(len(self)))
					self.close()
				else:
					logging.info("[NEXT {}] Price diff small to close long position. Skipping".format(len(self)))

			elif(self.position.size==0):
				logging.info("[NEXT {}] Creating sell order".format(len(self)))
				logging.info("[NEXT {}] Order Size: {}".format(len(self), order_size))
	
				self.order = self.sell(size=order_size)


	def notify_order(self, order):


		logging.info('[NOTIFY_ORDER {}] ORDER {} - {}'.format(len(self), order.Status[order.status], order.tradeid))

		if order.status in [order.Submitted]:
			# Buy/Sell order submitted/accepted to/by broker - Nothing to do
			return


		if order.status in [order.Accepted]:
			# Buy/Sell order submitted/accepted to/by broker - Nothing to do			
			return

		if order.status in [order.Expired, order.Cancelled, order.Rejected]:
			
			pass

		if order.status in [order.Completed]:

			order_type  = "buy" if( order.isbuy() ) else "sell"

			logging.info('[NOTIFY_ORDER {}] {} ORDER EXECUTED [{}], Size: {}, Price: {}, Cost: {}, Comm {}'.format(
				len(self),
				order_type.upper(), 
				order.ref,
				order.executed.size,
				order.executed.price,
				order.executed.value,
				order.executed.comm))

		self.order = None


	def notify_trade(self, trade):

		self.order = None

		if not trade.isclosed:
			return

		trade_info = {
			"tradeid"  : trade.tradeid,
			"traded_price" : trade.price,
			"trade_value"   : trade.value,
			"trade_gross" : trade.pnl,
			"trade_net" : trade.pnlcomm,
			"bars"  :len(self)
		}

		logging.info('[NOTIFY_TRADE] OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm))

	def stop(self):

		 
		final_analysis = [
			("strategy"		, os.path.basename(__file__)),
			("timeframe_0_tf", self.p.timeframe_0_tf), 
			("value"     , self.broker.get_value()),
			("cash"      , self.broker.get_cash()),
			# ("drawdown"  , dict_recursively(self.analyzers.myTimeDrawDown.get_analysis())),
		]

		final_analysis = collections.OrderedDict(final_analysis)


		str_to_format = "{}," * len(final_analysis.keys())            
		print(str_to_format.format(*final_analysis.values()) ,end="")

backtrader

Even if it obvious to you, the fragments above don't show how many orders you send to the market or even the cash being set to 100k and many other things and not even Order.Margin being notified (and which is not even considered in notify_order)

Is every order being met with Order.Margin? Are some orders going through? The usual chart with the evolution of the cash and account value can already reveal a lot (it usually includes also all the buy and sell points)

You were asking:

@Taewoo-Kim said in Why skip order.Margin status?:

Is there a quick way to debug this?

Yes. Add some simple print statements just before and after this:

@Taewoo-Kim said in Why skip order.Margin status?:

		if(self.buysig is None or self.sellsig is None):
			return

And you will see for example why that statement means nothing.

These statements are also meaningless:
@Taewoo-Kim said in Why skip order.Margin status?:

		self.buysig = bt.ind.CrossDown(self.datas[0].close[0], self.sma)
		self.sellsig = bt.ind.CrossUp(self.datas[0].close[0], self.sma)

Using [x] (on close) during the init phase is only giving you a fake value because you have preloaded the data. Disable preloading with cerebro.run(preload=False) and you will see it explode, because there will be no data there.

That means you are generating absolutely random buy and sell signals which are probably kicking in very often given the use of a fixed value in the comparison against the moving average.

Taewoo Kim

Setting up the cerebro

config_tf = conf_ini["timeframes"]["timeframe_0_tf"]
timeframe, compression = pandas_tf_to_backtrader_tf(config_tf)

midpoint, _, csv = get_data(
currency=conf_ini["general"]["currency"],
num_days_to_lookback=args.num_days_to_lookback,
resample=config_tf
)

timeframes = []
cerebro = bt.Cerebro(maxcpus=(int(conf_ini["general"]["maxcpus"]) if conf_ini["general"]["maxcpus"] is not "None" else None))

bt_feed = bt.feeds.GenericCSVData(
dataname=csv,
timeframe=timeframe,
datetime=0,
open=1,
high=2,
low=3,
close=4,
volume=5,
openinterest=-1
)

cerebro.adddata(bt_feed)

cerebro.addobserver(bt.observers.Broker)
cerebro.addobserver(bt.observers.Trades)
cerebro.addobserver(bt.observers.BuySell)

cerebro.addstrategy(St,
timeframe_0_tf = conf_ini["timeframes"]["timeframe_0_tf"],
currency=conf_ini["general"]["currency"]
)

cash = convert_data_type(conf_ini["broker"]["cash"])
cerebro.broker.setcash(cash)

stakesize = convert_data_type(conf_ini["broker"]["size"])
cerebro.addsizer(bt.sizers.FixedSize, stake=stakesize)

commission = convert_data_type(conf_ini["broker"]["commission"])
leverage = convert_data_type(conf_ini["broker"]["leverage"])

cerebro.broker.setcommission(commission=commission, leverage=leverage)

cerebro.addanalyzer(btanalyzers.SharpeRatio, _name='mySharpeRatio', timeframe=bt.TimeFrame.Years, compression=1, riskfreerate=0.0)
cerebro.addanalyzer(btanalyzers.DrawDown, _name='myDrawDown')
cerebro.addanalyzer(btanalyzers.AnnualReturn, _name='myAnnualReturn')
cerebro.addanalyzer(btanalyzers.TimeDrawDown, _name='myTimeDrawDown')

cerebro.addanalyzer(btanalyzers.TradeAnalyzer, _name='myTradeAnalyzer')

cerebro.addanalyzer(btanalyzers.GrossLeverage, _name='myGrossLeverage')

from datetime import datetime
timestarted = datetime.now()
thestrats = cerebro.run()
#strats = [s[0] for s in thestrats] # flatten the result

for i, strat in enumerate(thestrats):
```
 final_stats = {}
 parts=[]

 drawdown = dict_recursively(strat.analyzers.myTimeDrawDown.get_analysis())
 sharpe = dict_recursively(strat.analyzers.mySharpeRatio.get_analysis())
 annual = dict_recursively(strat.analyzers.myAnnualReturn.get_analysis())
 annual_return =  next (iter (annual.values())) #
 my_analyis = [
     ("sharperatio" , sharpe["sharperatio"]),
     ("maxdrawdown" , drawdown["maxdrawdown"]),
     ("maxdrawdownperiod" , drawdown["maxdrawdownperiod"]),
     ("annual_return" , annual_return),
     ("time_taken"    , int((datetime.now() - timestarted).total_seconds()))
 ]
 my_analyis = collections.OrderedDict(my_analyis)

 string_to_format="{}," * len(my_analyis.values())
 print(string_to_format.format(*my_analyis.values()))
```
#cerebro.plot(style='bar')
cerebro.plot()
Yes, every LONG order is being met with Order.Margin.
How many orders:

fragments above don't show how many orders you send to the market

What I wanted to convey in the code was that I wanted to have only ONE order and wait for it until it is accepted. Is this the wrong logic?

Missing Order.Margin

not even Order.Margin being notified (and which is not even considered in notify_order)

My apologies... i copy/pasted an older version of notify_order. It should be this (but doesn't matter b/c it just passes):

def notify_order(self, order):


	logging.info('[NOTIFY_ORDER {}] ORDER {} - {}'.format(len(self), order.Status[order.status], order.tradeid))

	if order.status in [order.Submitted]:
		# Buy/Sell order submitted/accepted to/by broker - Nothing to do
		return


	if order.status in [order.Accepted]:
		# Buy/Sell order submitted/accepted to/by broker - Nothing to do			
		return

	if order.status in [order.Expired, order.Cancelled, order.Rejected]:
		
		pass

	if order.status in [order.Margin]:
		# forgot to include this
		pass
	

	if order.status in [order.Completed]:

		order_type  = "buy" if( order.isbuy() ) else "sell"

		logging.info('[NOTIFY_ORDER {}] {} ORDER EXECUTED [{}], Size: {}, Price: {}, Cost: {}, Comm {}'.format(
			len(self),
			order_type.upper(), 
			order.ref,
			order.executed.size,
			order.executed.price,
			order.executed.value,
			order.executed.comm))

	self.order = None

signal on init

That means you are generating absolutely random buy and sell signals which are probably kicking in very often given the use of a fixed value in the comparison against the moving average.

So do i need to calculate the signals within next() ?

Taewoo Kim

Here's part of the output from stdout when I added bunch of logging statements.
The first order (which happened to be long) gets order.Margin status

[INFO] 16:17:25 RUNNING LOCAL BACKTEST
[INFO] 16:17:25 Reading cached pickle cache/oandalabs-GBP_JPY-2017-06-28_30days_15Min.pkl
[INFO] 16:17:25 *       *       *       *       *       *       *       *       *       *       *       *       *       **       *       *       *       *       *       *       *       *       *       *       *       *       *       *       *
[INFO] 16:17:25 Running backtest - 4 15
[INFO] 16:17:25 MaxCPUs 8
[INFO] 16:17:25 Adding strategy
[INFO] 16:17:25 Cash 100000
[INFO] 16:17:25 Cash 100000
[INFO] 16:17:25 Stake Size 2000
[INFO] 16:17:25 Stake Size 2000
[INFO] 16:17:25 Commission 0.0
[INFO] 16:17:25 Commission 0.0
[INFO] 16:17:25 Leverage 10
[INFO] 16:17:25 Leverage 10
[INFO] 16:17:25 Running cerebro

[INFO] 16:17:26 [NEXT 90] Creating buy order
[INFO] 16:17:26 [NEXT 90] Order Size: 7804.36029609743
[INFO] 16:17:26 [NOTIFY_ORDER 91] ORDER Submitted - 0
[INFO] 16:17:26 [NOTIFY_ORDER 91] ORDER Margin - 0

backtrader

@Taewoo-Kim said in Why skip order.Margin status?:

So do i need to calculate the signals within next() ?

The calculation during __init__ is not done on single point value (i.e.: [x]) of the line object, but on the line itself.

buysig = self.data > self.data1
# or
buysig = bt.ind.CrossOver(self.data, sma)
# or more specific
buysig = bt.ind.CrossOver(self.data.close, sma)

Because [x] is a construct which isn't valid during __init__. See:

Docs - Platform Concepts (probably the section about Operators, with Stage1 and Stage2

backtrader

@Taewoo-Kim said in Why skip order.Margin status?:

Is this the wrong logic?

Yes. self.buysig and self.sellsig are objects which you have instantiated yourself during __init__ so they cannot be None. And given the creation of the signals is against a fixed value, they will evaluate to True at random times (random from the perspective that it is unknown how the data looks like)

A very simple but effective logic is that of the sentinel (self.order) shown in the Docs - Quickstart which you have already seen.

As to why you run into Order.Margin, it is really impossible to say. The code is incomplete and somethings have changed from the 1st post to now. There are things which seem odd like setting maxcpus (optimization?)

Things you may do:

Set check_submit=False in the broker.

This may simply delay the problem to the execution moment
Print the generated order (print(order)) to see the generated details
Print the cash level before generating an order
Simplify your case by removing all the unnecessary stuff (analyzers, custom observer configuration, loading data from conf, broker configuration) and keeping just the basic logic.

Taewoo Kim

@backtrader

First off, thank you for all the help you've given me (and everyone else for that matter). BT framework is definitely more powerful than I originally thought.

I figured out that it was the currency issue - mainly JPY ( i was using GBP_JPY). When BT multiplied size of order (calculated in USD as per Oanda's formula) by the "per share" cost of GBP_JPY unit, it exceeded the cash balance. Had to add exception to divide size by 100 if JPY is quoted currency.

Thanks for your help once again! Sorry for making you chase the wild goose

Taewoo Kim

Before i close off this thread.. do you see anything wrong with this syntax? Im getting zero trades...

def __init__(self):

	self.fast_ema = bt.ind.ExponentialMovingAverage(period=8, plotname="Fast EMA")
	self.slow_ema = bt.ind.ExponentialMovingAverage(period=26, plotname="Slow EMA")		
	self.stddev = bt.ind.StandardDeviation(period=8)	

	self.regime = self.fast_ema - self.slow_ema    # Positive when bullish

	self.buysig = bt.ind.And(
				self.fast_ema > self.datas[0].close, 
				self.datas[0].close >  self.slow_ema,
				bt.ind.CrossUp(self.datas[0].close, self.fast_ema),
				self.regime > 0
			)

	self.sellsig = bt.ind.And(
				self.fast_ema < self.datas[0].close,
				self.datas[0].close <  self.slow_ema,
				bt.ind.CrossDown(self.datas[0].close, self.fast_ema),
				self.regime < 0
			)
	self.order = None

backtrader

The syntax is ok. But the logic seems doomed to fail:

@Taewoo-Kim said in Why skip order.Margin status?:

	self.buysig = bt.ind.And(
				self.fast_ema > self.datas[0].close, 
				self.datas[0].close >  self.slow_ema,
				bt.ind.CrossUp(self.datas[0].close, self.fast_ema),
				self.regime > 0
			)

  		self.fast_ema > self.datas[0].close, 
  		self.datas[0].close >  self.slow_ema,

These 2 place the close in between the fast_ema and slow_ema

  		bt.ind.CrossUp(self.datas[0].close, self.fast_ema),

And This one says that the close should have just crossed the fast_ema to the upside. Which looking at the previous statements is not possible (not possible at the same time)

Why skip order.Margin status?

cerebro.addanalyzer(btanalyzers.TradeAnalyzer, _name='myTradeAnalyzer')