For code/output blocks: Use ``` (aka backtick or grave accent) in a single line before and after the block. See: http://commonmark.org/help/

Memory leak?



  • Hi,

    While doing some testing on a large list of stocks, I noticed that my memory usage kept on climbing until it maxed out my ram. To simplify the results, I've used the code from the main page:

    Code:

    from __future__ import (absolute_import, division, print_function,
                            unicode_literals)
    
    import argparse
    import sys
    
    import backtrader as bt
    import backtrader.feeds as btfeeds
    import backtrader.indicators as btind
    import backtrader.utils.flushfile
    
    import resource
    import objgraph
    
    from datetime import datetime
    
    import matplotlib.pyplot as plt
    import time
    
    class SmaCross(bt.SignalStrategy):
            params = (('pfast', 10), ('pslow', 30),)
            def __init__(self):
                sma1, sma2 = bt.ind.SMA(period=self.p.pfast), bt.ind.SMA(period=self.p.pslow)
                self.signal_add(bt.SIGNAL_LONG, bt.ind.CrossOver(sma1, sma2))
    
    def backtrader_mem_check():
        data = bt.feeds.YahooFinanceData(dataname='MSFT', fromdate=datetime(2011, 1, 1),
                                         todate=datetime(2012, 12, 31))
    
        cerebro = bt.Cerebro()
        cerebro.adddata(data)
        cerebro.addstrategy(SmaCross)
        cerebro.run()
    
        plt.ion()
        cerebro.plot(numfigs = 10) #make problem worse by plotting 10 figs
        plt.close("all")
    
    def plot_mem_check():
        temp_list = []
        for a in range(1000000):
            temp_list.append(a)
        plt.plot(temp_list)
        plt.ylabel('some numbers')
        plt.ion()
        plt.show()
        plt.close("all")
    
    if __name__ == '__main__':
    
        print ('Memory usage START: %s (kb)' % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
        objgraph.show_most_common_types()
        print(" ")
    
        for a in range(100):
            if a != 0 and a % 25 == 0:
                print ('Memory usage MOD 25 CHECK: %s (kb)' % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
                objgraph.show_most_common_types()
                print(" ") 
            backtrader_mem_check()
    
        print ('Memory usage END: %s (kb)' % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
        objgraph.show_most_common_types()
        print(" ")
    

    Run:
    def backtrader_mem_check():

    Results: large increase in memory use

    Memory usage START: 68980 (kb)
    function           20678
    tuple              17295
    cell               8721
    dict               8594
    classmethod        7278
    weakref            5069
    list               4759
    type               3758
    OrderedDict        3746
    wrapper_descriptor 2146
     
    Memory usage MOD 25 CHECK: 1579512 (kb)
    dict                      965239
    list                      408549
    weakref                   405232
    _XYPair                   173200
    Path                      133171
    method                    107846
    Affine2D                  89645
    CompositeGenericTransform 72375
    MarkerStyle               71502
    Line2D                    71475
     
    Memory usage MOD 25 CHECK: 3053948 (kb)
    dict                      1922492
    list                      812645
    weakref                   805239
    _XYPair                   346400
    Path                      266321
    method                    215663
    Affine2D                  179334
    CompositeGenericTransform 144750
    MarkerStyle               143069
    Line2D                    142950
     
    Memory usage MOD 25 CHECK: 4527868 (kb)
    dict                      2878740
    list                      1216168
    weakref                   1205186
    _XYPair                   519600
    Path                      399471
    method                    323426
    Affine2D                  268986
    CompositeGenericTransform 217125
    MarkerStyle               214582
    Line2D                    214425
     
    Memory usage END: 5996092 (kb)
    dict                      3832047
    list                      1617972
    weakref                   1605124
    _XYPair                   692800
    Path                      532621
    method                    431064
    Affine2D                  358550
    CompositeGenericTransform 289500
    MarkerStyle               285970
    Line2D                    285900
    

    Run:
    def plot_mem_check():

    Results: stable memory use when plotting and closing windows using matplotlib

    Memory usage START: 69004 (kb)
    function           20678
    tuple              17295
    cell               8721
    dict               8593
    classmethod        7278
    weakref            5069
    list               4759
    type               3758
    OrderedDict        3746
    wrapper_descriptor 2146
     
    Memory usage MOD 25 CHECK: 954608 (kb)
    function           20732
    tuple              17141
    dict               11381
    cell               8805
    classmethod        7278
    weakref            5814
    list               5570
    OrderedDict        3758
    type               3756
    wrapper_descriptor 2146
     
    Memory usage MOD 25 CHECK: 957784 (kb)
    function           20804
    tuple              17261
    dict               15141
    cell               8917
    classmethod        7278
    weakref            6750
    list               6650
    OrderedDict        3774
    type               3756
    wrapper_descriptor 2146
     
    Memory usage MOD 25 CHECK: 1004780 (kb)
    function           20714
    tuple              17111
    dict               10441
    cell               8777
    classmethod        7278
    weakref            5580
    list               5300
    type               3756
    OrderedDict        3754
    wrapper_descriptor 2146
     
    Memory usage END: 1004780 (kb)
    function           20777
    tuple              17216
    dict               13731
    cell               8875
    classmethod        7278
    weakref            6399
    list               6245
    OrderedDict        3768
    type               3756
    wrapper_descriptor 2146
    

    This problem isn't noticeable when running backtrader if plotting is turned off. Any ideas?

    Thanks


  • administrators

    I fail to understand what your expectation is or why you think that a memory leak is there.

    Python (actually CPython) uses reference counting before garbage collecting things. Until you leave the scripts, there will be thousands of references which are still alive.



  • Hi,

    Thanks for the quick reply. Pardon my lack of knowledge with Python, just started using it.

    I was asking the question because when I run "plot_mem_check" I see a saw tooth pattern in my memory use and it never exceeded a certain ceiling.

    I looked into manual garbage collection and it slowed down the amount of memory being used but didn't stop the growth. Is it possible to run backtrader to process a single stock at a time in a loop like above?

    Thanks



  • Hi,

    Got it figured out, had to create a process for each call to the backtrader analysis for each stock I want to check individually. Back to having a saw tooth memory pattern.

    Thanks


  • administrators

    If you were re-using cerebro, yes you were creating additional references to data feeds with each iteration and the number of references kept inside the engine would obviously grow.

    This has already been discussed in the past, Cerebro is not meant to be reused.


Log in to reply
 

});