@mrahmadt Running into an OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already for a list of 500 stocks. I have tried max_overflow=-1 and pool_size=0 to have unlimited connections. Still doesn't work. Anyone have any suggestions?
Hi all. I’ve been struggling with this problem for a bit now. The orders keep on failing on margin just a few trades in when a commission scheme is added.
In attempt to backtest more accurately, I added a commission scheme on the strategy working off of https://teddykoker.com/2019/04/backtesting-a-cross-sectional-mean-reversion-strategy-in-python/:
Things I have tried:
target=weights[i]/n Reason: instead of taking ~50% and ~50% of the portfolio size for each of the 2 currency pairs in the universe, I scale it way down to not use 100% of cash.cerebro.broker.set_shortcash(False). Reason: Shorting can count as cash added to an account.cerebro.broker.set_checksubmit(False) Reason: https://community.backtrader.com/topic/1611/having-margin-problem-when-reversing-the-position-using-order_target_percent/2. Similar problem, but for this strategy, doesn’t even do a partial order. Just straight to order failed because of margin call.cheat_on_open=True. Weights are calculated at close, so it’ll make sense to try and execute order on open to match.None of these solutions have worked for me, still margin called after the first few trades. Perhaps some of the attempts at solving this problem is not completely understanding config settings. It is the configuration that makes the most sense for me at this time.
Strategy:
class St(bt.Strategy):
def log(self, arg):
print("{} {}".format(self.datetime.datetime(), arg))
def __init__(self):
if self.p.backtest:
self.datastatus = 1
else:
self.datastatus = 0
def notify_data(self, data, status, *args, **kwargs):
print("*" * 5, "DATA NOTIF:", data._getstatusname(status), *args)
if status == data.LIVE:
self.datastatus = 1
def notify_order(self, order):
"""Run on every next iteration. Checks order status and logs accordingly"""
if order.status in [order.Submitted, order.Accepted]:
return
elif order.status == order.Completed:
if order.isbuy():
self.log(
"BUY EXECUTED price: {}, value: {}, commission: {}, broker: {}, cash: {}".format(
order.executed.price,
order.executed.value,
order.executed.comm,
self.broker.get_value(),
self.broker.get_cash(),
)
)
self.buyprice = order.executed.price
self.buycomm = order.executed.comm
else: # sell
self.log(
"SELL EXECUTED price: {}, value: {}, commission: {}, broker: {}, cash: {}".format(
order.executed.price,
order.executed.value,
order.executed.comm,
self.broker.get_value(),
self.broker.get_cash(),
)
)
elif order.status in [order.Margin, order.Rejected, order.Canceled]:
self.log(
"ORDER FAILED with status: {}, BROKER VAL: {}, CASH AVAILABLE: {}".format(
order.getstatusname(), self.broker.get_value(), self.broker.get_cash()
)
)
# change order variable back to None to indicate no pending order
self.order = None
def notify_trade(self, trade):
"""Run on every next iteration. Logs data on every trade when closed."""
if trade.isclosed:
self.log("CLOSE Gross P/L: {}, Net P/L: {}".format(trade.pnl, trade.pnlcomm))
def next(self):
# only look at data that existed yesterday
available = list(filter(lambda d: len(d), self.datas))
rets = np.zeros(len(available))
for i, d in enumerate(available):
# calculate individual daily returns
rets[i] = (d.close[0] - d.close[-1]) / d.close[-1]
# calculate weights using formula
market_ret = np.mean(rets)
weights = -(rets - market_ret)
weights = weights / np.sum(np.abs(weights))
for i, d in enumerate(available):
dt, dn = self.datetime.datetime(), d._name
if self.datastatus:
self.order_target_percent(d, target=weights[i] / 4)
self.log(
"ORDER CREATED: {:.2f} on {} with broker val: {} cash: {} weights: {} ".format(
d.close[0], dn, self.broker.get_value(), self.broker.get_cash(), weights[i] / 4
)
)
Commission Scheme:
class forexSpreadCommisionScheme(bt.CommInfoBase):
"""
https://backtest-rookies.com/2017/07/13/code-snippet-forex-commission-scheme/
"""
params = (
("spread", 2.0),
("JPY_pair", False),
("stocklike", False),
("acc_counter_currency", True),
("margin", 0.03), # highest margin requirements for these /USD pairs.
("leverage", 20.0), # 20x cash
("commtype", bt.CommInfoBase.COMM_FIXED),
)
def _getcommission(self, size, price, pseudoexec):
if self.p.JPY_pair == True:
multiplier = 0.01
else:
multiplier = 0.0001
if self.p.acc_counter_currency == True:
comm = abs((self.p.spread * (size * multiplier) / 2))
else:
comm = abs((self.p.spread * ((size / price) * multiplier) / 2))
return comm
Configuration:
cerebro = bt.Cerebro(quicknotify=True, cheat_on_open=True)
for ticker in ['NZD_USD', 'AUD_USD']:
# get datafeed here...
cerebro.adddata(data)
cerebro.broker.setcash(10_000)
cerebro.addstrategy(St)
cerebro.addsizer(bt.sizers.FixedSize, stake=1)
# Add the new commission scheme
comminfo = forexSpreadCommisionScheme(spread=2.0, acc_counter_currency=True)
cerebro.broker.addcommissioninfo(comminfo)
# Set slippage for realistic scenarios. Increase chance of order filling
cerebro.broker.set_slippage_fixed(0.0005, slip_open=False, slip_match=False, slip_out=False)
cerebro.broker.set_shortcash(False) # Adds cash when shorting, instead of subtracting
cerebro.broker.set_checksubmit(False) # Check margin/cash before accepting an order into the system
results = cerebro.run(tradehistory=True)
pnl = cerebro.broker.get_value() - args.cash
print("Profit or Loss: {:.2f}".format(pnl))
Logs:
2010-12-30 22:00:00 ORDER CREATED: 0.77 on NZD_USD with broker val: 10000.0 cash: 10000.0 weights: 0.12500000000000003
2010-12-30 22:00:00 ORDER CREATED: 1.02 on AUD_USD with broker val: 10000.0 cash: 10000.0 weights: -0.125
2010-12-30 22:00:00 BUY EXECUTED price: 0.77988, value: 24999.6, commission: 83.33200000000001, broker: 23011.875139999738, cash: 14091.601199999866
2010-12-30 22:00:00 SELL EXECUTED price: 1.02324, value: 24999.6, commission: 83.33200000000001, broker: 23011.875139999738, cash: 14091.601199999866
2010-12-31 22:00:00 ORDER CREATED: 0.78 on NZD_USD with broker val: 23011.875139999738 cash: 14091.601199999866 weights: 0.125
2010-12-31 22:00:00 ORDER CREATED: 1.02 on AUD_USD with broker val: 23011.875139999738 cash: 14091.601199999866 weights: -0.125
2010-12-31 22:00:00 CLOSE Gross P/L: -3041.618000000035, Net P/L: -3208.282000000035
2010-12-31 22:00:00 ORDER FAILED with status: Margin, BROKER VAL: 12298.29655999978, CASH AVAILABLE: 8325.02679999987
2010-12-31 22:00:00 ORDER FAILED with status: Margin, BROKER VAL: 12298.29655999978, CASH AVAILABLE: 8325.02679999987
2011-01-01 22:00:00 ORDER CREATED: 0.77 on NZD_USD with broker val: 12298.29655999978 cash: 8325.02679999987 weights: 0.125
2011-01-01 22:00:00 ORDER CREATED: 1.02 on AUD_USD with broker val: 12298.29655999978 cash: 8325.02679999987 weights: -0.125
2011-01-01 22:00:00 BUY EXECUTED price: 0.77328, value: 30745.199999999997, commission: 102.48400000000001, broker: 18258.873459999806, cash: 9794.544799999883
2011-01-01 22:00:00 ORDER FAILED with status: Margin, BROKER VAL: 18258.873459999806, CASH AVAILABLE: 9794.544799999883
2011-01-02 22:00:00 ORDER CREATED: 0.77 on NZD_USD with broker val: 18258.873459999806 cash: 9794.544799999883 weights: -0.12500000000000003
2011-01-02 22:00:00 ORDER CREATED: 1.02 on AUD_USD with broker val: 18258.873459999806 cash: 9794.544799999883 weights: 0.12499999999999997
2011-01-02 22:00:00 CLOSE Gross P/L: -6866.427999999926, Net P/L: -7071.395999999926
2011-01-02 22:00:00 ORDER FAILED with status: Margin, BROKER VAL: 32688.961160000075, CASH AVAILABLE: 15336.738800000057
2011-01-02 22:00:00 ORDER FAILED with status: Margin, BROKER VAL: 32688.961160000075, CASH AVAILABLE: 15336.738800000057
2011-01-03 22:00:00 ORDER CREATED: 0.76 on NZD_USD with broker val: 32688.961160000075 cash: 15336.738800000057 weights: -0.12499999999999978
2011-01-03 22:00:00 ORDER CREATED: 1.00 on AUD_USD with broker val: 32688.961160000075 cash: 15336.738800000057 weights: 0.12500000000000022
2011-01-03 22:00:00 SELL EXECUTED price: 0.75709, value: 81722.4, commission: 272.408, broker: 47940.29879999999, cash: 18939.106000000014
2011-01-03 22:00:00 ORDER FAILED with status: Margin, BROKER VAL: 47940.29879999999, CASH AVAILABLE: 18939.106000000014
2011-01-04 22:00:00 ORDER CREATED: 0.76 on NZD_USD with broker val: 47940.29879999999 cash: 18939.106000000014 weights: 0.12500000000000003
2011-01-04 22:00:00 ORDER CREATED: 1.00 on AUD_USD with broker val: 47940.29879999999 cash: 18939.106000000014 weights: -0.12499999999999997
....
....
....
2019-07-25 21:00:00 ORDER CREATED: 0.66 on NZD_USD with broker val: -94102645413.3025 cash: -94102645413.3025 weights: -0.12499999999999993
2019-07-25 21:00:00 ORDER CREATED: 0.69 on AUD_USD with broker val: -94102645413.3025 cash: -94102645413.3025 weights: 0.12500000000000008
2019-07-25 21:00:00 ORDER FAILED with status: Margin, BROKER VAL: -94102645413.3025, CASH AVAILABLE: -94102645413.3025
2019-07-25 21:00:00 ORDER FAILED with status: Margin, BROKER VAL: -94102645413.3025, CASH AVAILABLE: -94102645413.3025
2019-07-28 21:00:00 ORDER CREATED: 0.66 on NZD_USD with broker val: -94102645413.3025 cash: -94102645413.3025 weights: 0.125
2019-07-28 21:00:00 ORDER CREATED: 0.69 on AUD_USD with broker val: -94102645413.3025 cash: -94102645413.3025 weights: -0.125
2019-07-28 21:00:00 ORDER FAILED with status: Margin, BROKER VAL: -94102645413.3025, CASH AVAILABLE: -94102645413.3025
2019-07-28 21:00:00 ORDER FAILED with status: Margin, BROKER VAL: -94102645413.3025, CASH AVAILABLE: -94102645413.3025
2019-07-29 21:00:00 ORDER CREATED: 0.66 on NZD_USD with broker val: -94102645413.3025 cash: -94102645413.3025 weights: -0.125
2019-07-29 21:00:00 ORDER CREATED: 0.69 on AUD_USD with broker val: -94102645413.3025 cash: -94102645413.3025 weights: 0.125
Strategy run finished with Run ID: 19
Profit or Loss: -94102655413.30
Is there something I am overlooking/misunderstanding? Only in year 2011 of the logs are a handful of successfully executed orders and closed positions. Any insights as to what may be happening is appreciated and let me know if more information is required. Thank you!
@run-out Just tried it out. That's awesome! Thank you.
https://backtest-rookies.com/2017/08/22/backtrader-multiple-data-feeds-indicators/
Following the guide above, both in that blog and locally, the cerebro.plot outputs candlestick charts that have bars that are all red.
Searching up this display bug shows that it seems that the datafeed is corrupted of some sort. The datafeed files are available in the link above for recreation, but does anyone know how to fix barup and bardown colors?
Does it require a custom plot scheme? If so, are there any examples currently out there?
I was able to solve this by removing [i] altogether. After rereading the documentation from the link above:
"When we loop through the data feeds, we can simply pass that data feed object to the dictionary as a key reference to then access the indicators for that particular feed."
The ith index isn't needed as the reference to each datafeed is stored in the dth key value.
It would be nice to get confirmation if this understanding is correct though.
Following this extremely helpful guide to run strategy over multiple datafeeds: https://backtest-rookies.com/2017/08/22/backtrader-multiple-data-feeds-indicators/, I am having trouble accessing the different lines an indicator has to offer.
Indicator construction:
self.inds = dict()
for i, d in enumerate(self.datas):
self.inds[d] = dict()
self.inds[d]["bb"] = bt.indicators.BollingerBands(
d, period=self.params.BB_MA, devfactor=self.params.BB_SD)
self.inds[d]['sma2'] = bt.indicators.SimpleMovingAverage(
d.close, period=self.params.sma2)
next() method:
if self.inds[d]["sma2"][0] < self.params.SMA_VAL:
if (d.close[-1] > self.inds[d]["bb"].lines[i].top[-1]) and (
d.close[0] <= self.inds[d]["bb"].lines[i].top[0]
):
self.order = self.sell(data=d)
self.stopprice = self.inds[d]["bb"].lines[i].top[0]
self.closepos = self.buy(exectype=bt.Order.Stop, price=self.stopprice)
Going by my understanding thus far, self.inds[d]["bb"] can access the regular indicator. And then we can take that one step further by accessing the ith element of the current datafeed related to the line. But I get this error: if (d.close[-1] > self.inds[d]["bb"].lines[i].top[-1]) and ( AttributeError: 'LineBuffer' object has no attribute 'top'
I also tried self.inds[i][d]["bb"].lines.top[0] to no avail. With this the error becomes: KeyError: 0.
I am missing something such that the indicator line isn't being read properly in the strategy, but is read properly in the indicator itself?
class SlopeCalc(bt.Indicator):
lines = ('slopescalcs',)
params = dict(typeToCalc='volatility', period=5)
def __init__(self):
if self.params.typeToCalc == 'volatility':
self.lines.slopecalc = bt.talib.VAR(self.data.close, timeperiod=self.params.period, nbdev=1.0)
def next(self):
print("in ind", self.lines.slopecalc[0])
class LaneStrategy(bt.Strategy):
def log(self, txt, dt=None):
''' Logging function for this strategy'''
dt = dt or self.datas[0].datetime.date(0)
print('%s, %s' % (dt.isoformat(), txt))
def __init__(self):
# Keep a reference to the "close" line in the data[0] dataseries
self.dataclose = self.datas[0].close
self.slopevolat = SlopeCalc()
def next(self):
print("in strat",self.slopevolat[0])
Printing yields:
in ind 6.162160000044992e-07
in strat nan
in ind 5.590959997192613e-07
in strat nan
in ind 3.006904000013577e-06
in strat nan
in ind 8.901303999886423e-06
in strat nan
in ind 1.0513023999569882e-05
in strat nan
in ind 6.540775999708259e-06
in strat nan
in ind 4.6533039999019365e-06
in strat nan
in ind 1.3609624000210374e-05
in strat nan
in ind 2.923282400013605e-05
in strat nan
in ind 2.4948944000069417e-05
in strat nan
in ind 1.0268935999757645e-05
.....
@run-out Ah, yup. It's a data-feed problem. Using a different data-feed, everything works as expected. Will look into as to why the data-feed isn't working as expected. Thank you!
@run-out Still having a bit of trouble figuring this out. Still NaN for res. Maybe the Strategy implementation/inheritance is wrong as seen above?
@run-out That is very odd. Talib works since bt.talib.SMA is able to return values that isn't NaN and is set and plotted correctly if I just do `self.lines.upperlines = hsma. Perhaps calling the indicator class is wrong?
class BaseStrategy(bt.Strategy):
def __init__(self):
super().__init__()
self.ures = UpperLines()
class LinesStrategy(BaseStrategy):
params = (
('maperiod', 15),
)
def log(self, txt, dt=None):
''' Logging function for this strategy'''
dt = dt or self.datas[0].datetime.date(0)
print('%s, %s' % (dt.isoformat(), txt))
def __init__(self):
# Keep a reference to the "close" line in the data[0] dataseries
super().__init__()
self.dataclose = self.datas[0].close
def next(self):
super().next()
# Simply log the closing price of the series from the reference
self.log('Close, %.2f' % self.dataclose[0])
print(self.ures[0])
class UpperLines(bt.Indicator):
lines = ('upperlines',)
params = dict(atr_factor=1.5)
res_smooth = 14
res_period = 10
def __init__(self):
hsma = bt.talib.SMA(self.data.high, timeperiod=self.res_smooth)
res = bt.talib.MAX(hsma, timeperiod=self.res_period)
atr = bt.talib.ATR(self.data.high, self.data.low, self.data.close) * self.params.atr_factor
self.lines.upperlines = res
# TODO: figure out why this line doesn't like 'res'
# self.lines.upperlines = res + self.atr
With my current understanding, an indicator is a "line" which then can be played with. So I then would pass hsma into a MAX function. So my question is: how can I get the values from bt.talib.SMA to pass into the MAX function, which I believe is responsible for the NaN result of res?