Bug#329743: uses too much memory

[Colin Watson <cjwatson@debian.org>]

On Fri, Sep 23, 2005 at 08:31:43AM +0200, Joey Hess wrote:
> In d-i in a lowmem mode install (no translations), cdebconf uses 5 mb of
> ram. With partman running the next largest memory usage, by
> parted_server is only 1 mb (numbers for mipsel arch). This is largely
> because it keeps all the question and template info in memory, which is
> very wasteful.
> 
> We need a db backend that either stores template/question data in a form
> that can be memapped, or in a way that makes it easy to load in stuff
> on a per-question (or per-package) basis.

Making it mmap()able isn't just a matter of a new db backend (there's no
real reason that rfc822db couldn't be made to do that). The problem is
more fundamental, because this structure in the core isn't amenable to
mmap()ing:

struct template_l10n_fields
{
        char *language;
        char *defaultval;
        char *choices;
        char *indices;
        char *description;
        char *extended_description;
        struct template_l10n_fields *next;
};

My thought is that template_l10n_fields instances probably make up the
bulk of cdebconf's memory usage, and there are only a few functions that
access them directly. The set family are only called from the DATA
command implementation and from debconf-loadtemplate (the former we need
to save in cdebconf memory, but the latter can be loaded on-demand), and
the get family can just load the l10n fields on demand each time.

Making this reasonably efficient, then, would probably just require
writing a variant of rfc822_parse_stanza() which read from memory rather
than a file, and storing pointers in rfc822db to the start of the stanza
for each template in the mmap()ed templatedb, invalidating the pointers
where necessary.

I'll keep looking at all of this.

Cheers,

-- 
Colin Watson                                       [cjwatson@debian.org]