Re: python, python-gtk and memory usage

[Steve Greenland <stevegr@debian.org>]

On 30-May-01, 07:53 (CDT), Torsten Landschoff <torsten@debian.org> wrote: 
> I tried to reproduce the problem and I am not sure if I really see the
> bug. What I can say for sure is that memory is not given back to the
> kernel (the memory usage in top never drops when I free any data) but
> it seems like the reserved storage is reused for new data. 

While I can't speak for python in particular, typical behaviour for
C library based code is that malloc() checks its current pool for
the needed memory. If it is not available, it asks the kernel (via
(s)brk(2)) for more, and uses that. When the program releases the memory
via free(), the memory is marked as available in the malloc pool, but
is not automatically returned to the kernel, because the common case
is that the memory will be needed again (or the program is exiting, in
which case the memory will be returned again).

On of the issues is that the way the kernel "gives" memory to a process
is by extending its data segment. The way a process returns memory is
by truncating its data segment. The implication is that the process
can only return memory if the entire region is free. If you malloc 1
megabyte, and then malloc 1 byte, and that byte is after the megabyte,
and then free the megabyte, the program can't return any of it. (Yes,
that's a gross over-simplification because malloc() implementations
typically have multiple pools for different size allocations, but it
will do as an example, okay?)

> Interestingly when I run the following python commands interactively
> 
> >>> a = range(10000000)
> >>> a = None
> >>> a = range(10000000)
> 
> memory usage of the python process goes up to 154MB after the first command, 
> down to 116MB after the second (at which stage I thought the bug is in
> python) but up to 154MB after the third command again so it is reusing the
> involved memory it seems.

What was the allocation before you issued the first command? What if you do
something like:

 >>> a = range(10000000)
 >>> b = range(10000)
 >>> a = None

[Steve actually looks at the bug report]:

Ahhh, yes, I remember some other bug with a similar
story. In particular, there was a sequence that of
gtk_style_new()/append()/clear() that looked reasonable, but doesn't
actually work because of the way they worked internally: there was an
extra reference created and not deleted. That one was in C, though, so
may not apply.

I'm not a python person, but I wouldn't be surprised if this code
sequence created an extra reference via t[0]:

 def add10000_clist(_button, clist=clist, t=text):
            clist.freeze()
            for i in range(10000):
                t[0] = "Row "+str(i)
                clist.append(t)
            clist.thaw()

(I wouldn't be surprised if it didn't, though. Again: not a python
person.)

HTH,
Steve

--
Steve Greenland <stevegr@debian.org>
(Please do not CC me on mail sent to this list; I subscribe to and read
every list I post to.)