Or open a file with standard Python code and write out to it.

@sagittarius19 said in How to filter large stock universe then run trading system on small group: @backtrader thanks for the help regarding this. I know its prolly a very simple fix. Here is what I updated to, and im still getting an error. class Pandasclass(btfeeds.PandasData): lines = ('symbol') params = (('symbol', -1),) btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol'] File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 13, in <module> class Pandasclass(btfeeds.PandasData): File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 364, in __new__ linesoverride, lalias=la) File "c:\Python36\lib\site-packages\backtrader\lineseries.py", line 130, in _derive clslines = baselines + lines TypeError: can only concatenate tuple (not "str") to tuple When i put another dummy value next to it, I got this error. class Pandasclass(btfeeds.PandasData): lines = ('symbol','blank') params = (('symbol', -1),('blank', -1),) btfeeds.PandasData.datafields = btfeeds.PandasData.datafields + [u'symbol', u'blank'] File "c:\Python27_path\Python27 Scritps\Extended pandas import attempt4.py", line 49, in <module> cerebro.run() File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) File "c:\Python36\lib\site-packages\backtrader\cerebro.py", line 1209, in runstrategies data.preload() File "c:\Python36\lib\site-packages\backtrader\feed.py", line 435, in preload while self.load(): File "c:\Python36\lib\site-packages\backtrader\feed.py", line 476, in load _loadret = self._load() File "c:\Python36\lib\site-packages\backtrader\feeds\pandafeed.py", line 255, in _load line[0] = self.p.dataname.iloc[self._idx, colindex] File "c:\Python36\lib\site-packages\backtrader\linebuffer.py", line 222, in __setitem__ self.array[self.idx + ago] = value TypeError: must be real number, not str Does datas not allow for non numerical values to be in the columns? All values are floats. This is not a database.

@algoguy235 said in Detect new resampled price: I wasn't able to get len() as a line object in the init function len is a builtin and cannot be overridden. Given it is only meant to check a single value (because it increase monotonically) it seems pointless to have a replacement function for it to deliver a lines object.

@backtrader any suggestions?

Probably because you have a wrong understanding of what a Stop order is. It is not a stop-loss, it is a Stop, it may sell with a profit. When a trigger price is met, a Market order is executed. Unless you check the execution prices, both buy and sell it is pointless to make assumptions about a Trade being wrong. For the meaning of each order type, see: Docs - Orders Docs - Order Management and Execution

@aarzt93 said in Question regarding Fractal Indicator: But when I print it out later it gives me all the high prices instead of only those highs that were marked with a bearish fractal. @aarzt93 said in Question regarding Fractal Indicator: if self.fractal.fractal_bearish: self.__special_array_for_later.append(self.high[-2]) @backtrader said in Question regarding Fractal Indicator: Any value which is not an indication will be a NaN because there is nothing there. It was clearly stated that values which are not marked as fractal are NaN, so you should be specifically looking to skip NaN values. In any case the logic is flawed, because the mark for a fractal doesn't happen now, it happens two periods in the past, exactly as the high you want to store. @aarzt93 said in Question regarding Fractal Indicator: okay, so is there a way to store all the bearish/bullish fractals in a container (dict/array) and have them printed out, so that I can gather some statistics on them ? They are already stored in the indicator itself, which is a line. You probably want to read the documentation here: Docs - Platform Concepts and look for Slicing

Aaah got it thanks

@jmoons Thanks. I have experienced the same issue again after I added "bound off=1" to 15mins bar resampling. Not sure whether the parameter setting change is the direct cause though. Another suspicion is paper account. So far I just did testing using ib paper account and I've noticed the paper account server has different maintenance schedule from ib production server.

Thank you for point out the problem.

@backtrader Fantastic, thank you!

What an adventure. A lot of time spent and the solution is quite elegant. @backtrader was right of course; notify_timer was the key. Send me a PM if you would like the solution. It's rather simple, but ya, learned from reading the documentation more, reading what backtrader said and then some furthering of my python ability.

@backtrader ohoh I get it. My English understanding is really poor. Thanks for your patience.

I found a way to solve the error, in this thread: https://community.backtrader.com/topic/1043/dates-and-time-loaded-from-csv-files-not-precise/2

After some more working on it, I feel I am close to working through the problem (just have to wait for markets to open again on sunday night). When I run my latest script, I receive this error: TWS Time at connection:20190111 15:00:43 EST Traceback (most recent call last): File "D:/Dropbox/Parker Magic Dropbox/Elliot Parker/Drums/QuantStrategyProjectFiles/__init__ (1) (1) (1).py", line 222, in <module> run() File "D:/Dropbox/Parker Magic Dropbox/Elliot Parker/Drums/QuantStrategyProjectFiles/__init__ (1) (1) (1).py", line 219, in run cerebro.run() File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1127, in run runstrat = self.runstrategies(iterstrat) 2018-12-12 AAPL-STK-SMART-USD File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1298, in runstrategies not pos self._runnext(runstrats) tradetoday finish File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\cerebro.py", line 1630, in _runnext strat._next() File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\strategy.py", line 325, in _next super(Strategy, self)._next() File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\lineiterator.py", line 270, in _next self.prenext() File "D:/Dropbox/Parker Magic Dropbox/Elliot Parker/Drums/QuantStrategyProjectFiles/__init__ (1) (1) (1).py", line 110, in prenext self.next() File "D:/Dropbox/Parker Magic Dropbox/Elliot Parker/Drums/QuantStrategyProjectFiles/__init__ (1) (1) (1).py", line 116, in next pos = self.getposition(d).size File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\strategy.py", line 1366, in getposition return broker.getposition(data) File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\brokers\ibbroker.py", line 306, in getposition return self.ib.getposition(data.tradecontract, clone=clone) File "C:\Users\Elliot Parker\Anaconda3\lib\site-packages\backtrader\lineseries.py", line 461, in __getattr__ return getattr(self.lines, name) AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute 'tradecontract' So it is able to recognise the first datafeed, but is unable to recognize the second datafeed's name and date, this I am not sure how to solve. I might try cleaning up that file as so it only has one of each date, sorted, etc. My latest script: import datetime import backtrader as bt from http.cookiejar import _debug from Functions.PandasData_Extend import PandasData_Extend import pandas as pd import pytz # Create a Stratey class ADRLeader(bt.Strategy): StrategyInformation = dict( StrategyName='NextDayStrategyBinsS', Notes='US Leads AUS, BHP, Bin 0, Short| Freebandz', OrderExecutionType='Market', BuyLogic='%Bracket', SellLogic='%Bracket', DateRun=datetime.datetime.now(), DataGranularity='Day' ) params = dict( ma=bt.ind.SMA, p1=5, p2=15, limit=0.005, limdays=1, limdays2=1, hold=1, usebracket=False, # use order_target_size switchp1p2=False, # switch prices of order1 and order2 limu=0.005, limd=0.01, valid=1, when=bt.timer.SESSION_START, bin=0, ) def __init__(self): self.add_timer(when=self.p.when) self.orefs = list() self.order = None self.o = dict() self.signal = self.datas[1].PercentileDailyPctChange self.tradetoday = 0 def notify_data(self, data, status, *args, **kwargs): # print('*' * 5, 'DATA NOTIF:', data._getstatusname(status), *args) if status == data.LIVE: self.datastatus = 1 def notify_store(self, msg, *args, **kwargs): # print('*' * 5, 'STORE NOTIF:', msg) return def notify_order(self, order): if order.status == order.Completed: # self.holdstart = len(self) # print('Order Completed') # print(self.datas[0].datetime.datetime()) # print(order.price) return if order.status == order.Accepted: # print('Order Accepted') return if not order.alive() and order.ref in self.orefs: self.orefs.remove(order.ref) if order.isbuy(): if order.status == order.Submitted: # print('Order Submitted BUY') # print(order.price) # print(str(order.price) + 'Order Buy Price') return if order.issell(): if order.status == order.Submitted: # print('Order Submitted SELL') # print(order.price) return def notify_trade(self, trade): if trade.isclosed: # print('tradeclosed') # date = self.datas[0].datetime.datetime() # print(self.datas[0].datetime.datetime()) # print(date, trade.pnl, trade.pnlcomm) # tradetoday = (bt.num2date(trade.dtopen)).date() # print(trade.ref) # self.listforall.append(tradetoday) # print('tradetoday') # print(tradetoday) self.tradetoday = 1 if trade.isopen: # print('tradeopen') # date = self.datas[0].datetime.datetime() # time = self.datas[0].datetime.time(0) # print(date, trade.price) return def notify_timer(self, timer, when): # print(str(when.time()) +' '+' juice') if when.time() == datetime.time(0,0): # print('Its a new day') # print(self.datas[0].datetime.datetime()) self.tradetoday = 0 else: # print('does not') return def prenext(self): self.next() def next(self): for i, d in enumerate(self.datas): dt, dn = self.datetime.date(), d._name pos = self.getposition(d).size # print(AAPL-STK-SMART-USD) print('{} {}'.format(dt, dn)) # print(i[0]) #Setting up the Not in Position if i == 0: pos = self.getposition(d).size # print('position size start') # print(pos) # print('position size finish') else: pass if pos == 0: print('not pos') #Setting up the If Not traded today if not self.tradetoday == 1: # print('tradetoday start') # print(self.tradetoday) print('tradetoday finish') if (len(self.datas[i]) > 0): # print(self.datetime.date()) #print(self.signal) print('top') # print(self.data0.Close) print('bottom') # print (self.data0.l.close[-1]) if d._dataname.PercentileDailyPctChange[+1] == 12: print('PercentileValue') print(d.datetime.datetime().isoformat()) print(d._dataname.PercentileDailyPctChange[+1]) print('PercentileValueFinish') if i == 0: print(d.close[0]) # self.buy(data=d, ) elif i == 0: print('0') print('Currency Latest Close') print(self.datas[0].datetime.datetime().isoformat()) print(d.close[0]) print('Currency Latest Finish') else: print('nope') return else: print('In Position') return else: print('no datasets coming in') return def start(self): if self.data0.contractdetails is not None: # print('yo') # print('Timezone from ContractDetails: {}'.format( # self.data0.contractdetails.m_timeZoneId)) return header = ['Datetime', 'Open', 'High', 'Low', 'Close'] # print(', '.join(header)) self.done = False def run(args=None): cerebro = bt.Cerebro() storekwargs = dict( host='127.0.0.1', port=7496, clientId=None, reconnect=100, timeout=10.0, _debug=False) ibstore = bt.stores.IBStore(**storekwargs) broker = ibstore.getbroker() cerebro.setbroker(broker) # 'EUR.USD-CASH-IDEALPRO' # AAPL-STK-SMART-USD # , tradename='AAPL-CFD-SMART-USD' # , tradename = 'BHP-CFD-SMART-AUD' # BHP-STK-SMART-AUD data0 = ibstore.getdata(reqRealTimeBars=True, dataname='AAPL-STK-SMART-USD', rtbar=False, timeframe=bt.TimeFrame.Seconds, compression=1) # cerebro.resampledata(data0, timeframe=bt.TimeFrame.Seconds, # compression=3) cerebro.adddata(data0) # print('remix') # print(data0) # cerebro.adddata(data0, name='d0') StockPercentile = pd.read_csv(r'D:\Dropbox\Parker Magic Dropbox\Elliot Parker\Drums\QuantStrategyResults\AUSLeadsUSPercentile.csv', header=0, index_col=0, parse_dates=True) StockPercentile = StockPercentile[~StockPercentile.index.duplicated(keep='first')] StockPercentile = StockPercentile.sort_index(ascending=False) StockPercentile = pd.DataFrame(StockPercentile.iloc[:, :]) # print(StockPercentile) data1 = PandasData_Extend(dataname=StockPercentile, timeframe=bt.TimeFrame.Days, compression=1) # cerebro.resampledata(data1) # cerebro.replaydata(data1) cerebro.adddata(data1) cerebro.addstrategy(ADRLeader) cerebro.run() if __name__ == '__main__': run()

I was able to find a solution, self.data.close.get(size=1, ago=-1)[0] works right as I want

@noobtrader nothing in your post shows that you are using live data. So why do you expect to have it in the answer?

@backtrader if you don't mind I would correct your summary slightly: @ab_trader is looking to use the signals and position sizes provided by bt, and mirror them to broker account by hands. bt will keep position sizes, cash, value and commissions. In that scenario there is no need to load the positions, cause they are mirrored by trader and are the same in bt and at broker account. In case of the discrepancies bt strategy can be easily adjusted. The broker is being used for signal and position sizes providing as well as for tracking of the cash and value (no need to support the real broker). I believe this summary reflects my proposals better. :)

One last question on this, as mentionned, I did use a csv file with the exact same column structure as in your example (columns=[bid, ask, datetime in last position]). The original file however come with the following structure [symbol, datetime, bid, ask] as show here: EUR/USD,20180201 00:00:00.125,1.24171,1.24173 EUR/USD,20180201 00:00:00.262,1.24172,1.24173 EUR/USD,20180201 00:00:00.695,1.24172,1.24175 EUR/USD,20180201 00:00:00.838,1.24173,1.24178 EUR/USD,20180201 00:00:00.848,1.24174,1.24177 Thus modifing the BidAskCSV class: class BidAskCSV(btfeeds.GenericCSVData): linesoverride = True # discard usual OHLC structure # datetime must be present and last lines = ('PAIR','datetime','bid', 'ask') # datetime (always 1st) and then the desired order for params = ( ('PAIR', 0), # inherited from parent class ('datetime', 1), # inherited from parent class ('bid', 2), # default field pos 1 ('ask', 3) # default field pos 2 ) The above won't work due to lines = ('PAIR','datetime','bid', 'ask') however replacing the section below instead is not returning an error message and so seems to be working fine: lines = ('datetime','bid', 'ask') I understand that the field 'PAIR' being a string is what is causing the error (code section below from -loadline), thus my questions: What is the underlying rationale ? Is it because "lines" should only refers to preset categories defined in BT (datetime, OHLV, volume, etc...)? What is the impact of simply not mentioning "PAIR" in the "lines" list (if any)? for linefield in (x for x in self.getlinealiases() if x != 'datetime'): # Get the index created from the passed params csvidx = getattr(self.params, linefield) if csvidx is None or csvidx < 0: # the field will not be present, assignt the "nullvalue" csvfield = self.p.nullvalue else: # get it from the token csvfield = linetokens[csvidx] if csvfield == '': # if empty ... assign the "nullvalue" csvfield = self.p.nullvalue # get the corresponding line reference and set the value line = getattr(self.lines, linefield) line[0] = float(float(csvfield))# Why is the expectation to ALWAYS have a float - can it be changed for more flexibility? (cg 'PAIR' filed in TruFX import - Error message return True

@revelator101 with regards to pyfolio, the sample that is in the sources run seamlessly against pyfolio A screenshot of today's run:

Hi, If your number 1 goal is: Learn Python so I can make use of Python to code my strategy You shouldn't probably be using btrun which is meant to abstract things and won't help with learning. Non code reports are difficult to look into. In your case the last lines give an insight: File "c:\users\charl\anaconda3\lib\site-packages\backtrader\feeds\btcsv.py", line 44, in _loadline y = int(dttxt[0:4]) ValueError: invalid literal for int() with base 10: '14/0' You are loading a csv data with the BactraderCSVData data feed and the field which is being encountered is 14/0 which cannot be parsed. If that csv file is ok you best options: Use GenericCSVData (see here) and configure the fields according to reality Load it with pandas.read_csv and then use a PandasData (see here)data feed also configuring the fields But if you are learning I would recommend focusing on easy and tested samples (and the quickstart), rather than feeding in your own data sources.