Python memory leak

Useful tools

resource

‘resource’ module for finding the current (Resident) memory consumption of your program
[Resident memory is the actual RAM your program is using]

>>> import resource

>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

4332

objgraph

‘objgraph’ is a useful module that shows you the objects that are currently in memory
[objgraph documentation and examples are available at: https://mg.pov.lt/objgraph/]

Lets look at this simple usage of objgraph:

import objgraph
import random
import inspect

class Foo(object):
   def __init__(self):
       self.val = None

   def __str__(self):
       return "foo - val: {0}".format(self.val)

def f():
   l = []
   for i in range(3):
       foo = Foo()
       #print "id of foo: {0}".format(id(foo))
       #print "foo is: {0}".format(foo)
       l.append(foo)

   return l

def main():

   d = {}
   l = f()

   d['k'] = l

   print "list l has {0} objects of type Foo()".format(len(l))
   objgraph.show_most_common_types()

   objgraph.show_backrefs(random.choice(objgraph.by_type('Foo')),
                           filename="foo_refs.png"
                       )

   objgraph.show_refs(d, filename='sample-graph.png')

if __name__ == "__main__":
   main()

results are

list l has 3 objects of type Foo()
function                   6735 ### check it out
dict                       3182  ####check this out
tuple                      2885
wrapper_descriptor         2138 
list                       1998     #####check this out
weakref                    1642
method_descriptor          1241
getset_descriptor          1069
builtin_function_or_method 998
type                       815
Graph written to C:\Users\xxx\AppData\Local\Temp\objgraph-sw0a7bet.dot (5 nodes)
Image renderer (dot) not found, not doing anything else
Graph written to C:\Users\xxx\AppData\Local\Temp\objgraph-zvjwpdcp.dot (9 nodes)
Image renderer (dot) not found, not doing anything else

Notice that we are also holding 3182 instances of ‘dict’ in memory. Although we’ll come to that in a bit.

Memory reduction tips

Slots

Use Slots for objects that you have a lot of. Slotting tells the Python interpreter that a dynamic dict is not needed for your object (From the example in 2.2 above, we saw that each Foo() object had a dict inside it)

Defining your class with slots makes the python interpreter know that the attributes/members of your class are fixed. And can lead to significant memory savings!

Interning: Beware of interned strings!

Python remembers immutables like strings (upto a certain size, this size is implementation dependent). This is called interning.

>>> t = “abcdefghijklmnopqrstuvwxyz”
>>> len(t)
26
>>> p = “abcdefghijklmnopqrstuvwxyz”
>>> id(t)
139863272322872
>>> id(p)
139863272322872

This is done by the python interpreter to save memory, and to speed up comparison. For eg, if 2 strings have the same id/reference – they are the same.

However, if your program creates a lot of small strings. You could be bloating memory.

Use format instead of ‘+’ for generating strings

Following from above, when building strings, prefer to build strings using format instead of concatentation.

That is,

st = "{0}_{1}_{2}_{3}".format(a,b,c,d)   # Better for memory. Does not create temp strs
   st2 = a + '_' + b + '_' + c + '_' + d # Creates temp strs at each concatenation, which are then interned.

In our system, we found significant memory savings when we changed certain string construction from concat to format.

To solve the memory leak problem , the following content is from stackoverflow

Here are the situations where the memory leak will happen in your python code

It depends on what kind of memory leak you are talking about. Within pure python code, it's not possible to "forget to free" memory such as in C, but it is possible to leave a reference hanging somewhere. Some examples of such:

an unhandled traceback object that is keeping an entire stack frame alive, even though the function is no longer running storing values in a class or global scope instead of instance scope, and not realizing it.
Cyclic references in classes which also have a del method. Ironically, the existence of a del makes it impossible for the cyclic garbage collector to clean an instance up.

poorly implemented C extensions, or not properly using C libraries as they are supposed to be.

Scopes which contain closures which contain a whole lot more than you could've anticipated

Default parameters which are mutable types:

import time
def foo(a=[]):
    a.append(time.time())
    return a

https://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html